codenote's blog

By codenote, history, 2 months ago, In English,

What is good approach to merge sorted files. I have around 10k files of size 1-2 mb and want to merge them in single sorted file.

 
 
 
 
  • Vote: I like it  
  • -2
  • Vote: I do not like it  

»
2 months ago, # |
  Vote: I like it +3 Vote: I do not like it

That sounds like you have only 20GB of data. If you have, say, 10GB, you can just read all the data in-memory and sort it. If you don't, you can do that thing with each, say, gigabyte you have and then merge resulting 20 files using several iterations of merge sort.

  • »
    »
    2 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    In linux i am not able to open more than 1024 files at a time.

    • »
      »
      »
      2 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Even i have set sysctl -w fs.file-max=500000 , then also it not opening. command used to set max files which we can open.

    • »
      »
      »
      2 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      You don't need to open more than two files at once. Open, fully read, close, repeat. Also, try looking at solutions here.