### nellex's blog

By nellex, history, 4 weeks ago,

Hi beautiful people of Codeforces! Hope you'd all be doing fine and keeping safe.

I'd like to start by mentioning that it's been a while since I graduated from university ever since which I haven't completely been able to indulge myself in competitive coding in the same way I did back in my university. But I sure haven't forgotten those days. I had some time off from work recently and decided to spend my time giving back to the community from where I have learned so much.

In the thought of so, I've built this tool which I like to call Harwest (a play on the word Harvest since the package name wasn't available anymore :'D) It's one of those tools which I would have loved to have during the time when I used to practice rigorously on this platform.

Harwest allows you to manage all the submissions that you've made on Codeforces, or on other OJs (WIP) into one single Git repository which you can then push to your personal Github / BitBucket / Gitlab or any other SCM platform of your choice.

Harwest completely automates this entire process with minimal effort from your side of answering just a few basic questions, and even goes to the limit of pushing the Git repository to the remote (Github/BitBucket..) for you!

Here's a sample repository built solely using Harwest: https://github.com/nileshsah/harwest-sample

The project directory structure is as follows:


nellex@HQ:~/accepted$tree . ├── codeforces │ ├── <contest-id> │ │ └── <problem-level> │ └── 592 │ └── A.cpp ├── README.md ├── submissions.json └── .git   Harwest creates a local git repository for you, scrapes all your submissions directly from Codeforces, maintains them in a nicely organized way, and commits them to the repository with the actual date of your submission! Now what the last statement enables us to do is to maintain a complete contribution graph on GitHub (or others) accurate to the time when you actually made the submission and not when you made the commit. Yes! It means complete tracking of your submissions on a day by day basis. Keep track of the number of days you've been lazy for, the number of problems you solved on a given day, and maintain a complete streak of daily practice! Heck, let this streak be your new motivation to practice continuously to become a better version of yourself every single day I've detailed the entire process of configuring Harwest on your local computer with an example in https://github.com/nileshsah/harwest-tool#installation Let me know if something needs more clarification. It's built using Python so you would require PIP with Python version 3.5 or above to install the package which you might have to set up in case you haven't used it before. It's a great package manager which I'd recommend learning a bit about nonetheless since it'll come in handy in the future. Along with PIP, you'd also need git on your computer which would come pre-installed if you're using Linux or MacOS though might would have to work your way up in case you're using windows. I'd also strongly encourage you to learn about Git since it's something that is heavily used in the software industry and early exposure to it would sure make your life a lot easier in the future. There are numerous documents on the internet which can help you get started on Git, one of them is https://www.freecodecamp.org/news/an-introduction-to-git-for-absolute-beginners-86fa1d32ff71/ Though For operating Harwest, you'd require minimal to no experience with Git. The first time Harwest runs, it'll scrape all your submissions starting from the first page till the last which might be a bit time-consuming if you had made a lot of submissions. If you don't want to parse the results till the very end, you can stop processing by hitting <Ctrl> + C. The next runs of Harwest will no more try to scrape the results from the old pages and instead will push your new changes directly. You can then make use of the harwest codeforces --start-page X feature to restart scraping from the page where you left off. I haven't got a chance to extensively test Harwest much and hence the term [Beta]. It seems to be working fine on MacOS and Ubuntu but I guess the auto push feature is causing a bit of an issue on Windows especially if you're using GitHub for Windows integration Do keep me posted if you face any other issues and I'd be happy to fix it at once. As for a workaround for the Windows user, if you want to leverage the automated push feature then specify the remote URL for your repository when asked as https://username:password@github.com/username/repository.git to bypass the login step during the push. Alternatively, you can always leave it empty and do a push from the generated repository by yourself. Nonetheless, I had a fun time building this tool and I really hope it would be helpful for you. Contributions to the project are more than welcome! ## TL;DR Automate the management of your submissions on GitHub with https://github.com/nileshsah/harwest-tool to get something like https://github.com/nileshsah/harwest-sample Update <2020-12-27>: Support for AtCoder is now available! Harwest now supports AtCoder! Massive thanks to s59_60r for leading the effort of finding the ideal way to integrate with the platform. Not to forget the crucial contributions from I_love_Hoang_Yen, YouKn0wWho, and others for fixing some major performance issues and making Harwest more stable. Much appreciated. The sample submissions repository https://github.com/nileshsah/harwest-sample has been updated to reflect what new changes this update brings. In order to switch to this release, please upgrade Harwest by running: pip install --upgrade harwest  Ensure that you're on version harwest-0.3.1, The new release is backward compatible and will not break any of your existing harvested submissions. After switching to the new version, you can then harvest submissions from AtCoder by running harwest atcoder  In addition to the new integration with AtCoder, we have introduced --full-scan capability on platforms to avoid the pain of dealing with re-starting harwest from the page where it stopped processing due to an error. It can be triggered as: harwest codeforces --full-scan OR harwest atcoder --full-scan  Please do let me know if you face any difficulties in the process, would be happy to help at once. Happy Harwesting! :) • +270  » 4 weeks ago, # | 0 Auto comment: topic has been updated by nellex (previous revision, new revision, compare).  » 4 weeks ago, # | ← Rev. 2 → 0 I installed Harwest but when I try to run the command.It says harwest: command not found.I’m using Ubuntu. •  » » 4 weeks ago, # ^ | ← Rev. 3 → +8 Hey sam571128! Thanks for trying. I'm guessing it might be so because the directory under which pip installed the package may not be present in your PATH. Could you try running the command pip3 uninstall harwest  nellex@HQ:~$ pip3 uninstall harwest Found existing installation: harwest 0.2.0 Uninstalling harwest-0.2.0: Would remove: /home/nellex/.local/bin/harwest /home/nellex/.local/lib/python3.5/site-packages/harwest-0.2.0.dist-info/* /home/nellex/.local/lib/python3.5/site-packages/harwest/* Proceed (y/n)? n  > /home/nellex/.local/bin/harwest The path /home/nellex/.local/bin is something you'd like to register under $PATH as export PATH=/home/nellex/.local/bin:$PATH or you can execute the command directly as /home/nellex/.local/bin/harwestLet me know if that works.
•  » » » 4 weeks ago, # ^ |   +5 Sorry for replying so late. But yea, it worked.
 » 4 weeks ago, # | ← Rev. 2 →   0 This is such a good idea! unfortunately I'm getting an error when running harwest codeforces (and putting in my handle): equest.py", line 1352, in do_open raise URLError(err) urllib.error.URLError: 
•  » » 4 weeks ago, # ^ | ← Rev. 6 →   0 Thanks a ton AkshajK I've updated the code to skip SSL verification. Can you try installing the new version by running pip3 install harwest==0.2.9 Let me know if the problem persists.
 » 4 weeks ago, # | ← Rev. 3 →   +19 Would like to extend my heartiest thanks to YouKn0wWho for helping me out in fixing some of the major encoding issues in Harwest on the windows platform. In case you were a windows user and have already tried Harwest then you might notice ^M characters in the generated submission files. If so, then I'd recommend deleting the locally created repository and starting over with harwest --init after updating to the latest version by running pip3 install harwest==0.2.9 Apologies for the inconvenience.As for YouKn0wWho, the amount of hard work you've put in this year itself is simply remarkable! https://github.com/ShahjalalShohag/ProblemSolving
 » 4 weeks ago, # |   0 I have installed it and pushed my submissions to github. Now if I make more submissions, how do I push them again so that it only takes the submissions which are not pushed yet?
•  » » 4 weeks ago, # ^ | ← Rev. 3 →   0 Hey anshu_2761! Thanks for giving Harwest a try! Harwest keeps track of your past submissions and fetches only the ones which are new. Whenever you'd like to sync the submissions in your GitHub repository with that on Codeforces, just run harwest codeforces and it'll do the job for you. Will also push automatically if you have set it up with auto-push else you can manually cd into the local submissions directory and issue a git push origin master yourself.I believe this was your question right? Also, I'd recommend running a pip3 install harwest==0.2.9 again since I pushed in some new set of fixes which would be good to pull.
 » 4 weeks ago, # |   +5 Nice tool!However, the process was stopping a lot before finishing all pages, and I had to use the commnad harwest codeforces --start-page x multiple times, then I wrote this python snippet to do the job. import subprocess for i in range(300): subprocess.call(['harwest', 'codeforces', '--start-page', str(i)]) 
•  » » 4 weeks ago, # ^ | ← Rev. 2 →   +13 Thanks Mohammad_Yasser! Nice hack I'd say! I would like to apologize that I didn't stress-test the tool enough to check its breaking limits. I_love_Hoang_Yen just helped me out in introducing a performance update (PR#3) which should address your issue a bit. Updating to the latest version of Harwest should help. Could you try executing the commands:  pip uninstall harwest pip install harwest  If it doesn't fetch the latest version of 0.2.10 then maybe you can do a force install using pip install harwest==0.2.10Let me know if that helps and thanks for trying Harwest out :)
•  » » » 4 weeks ago, # ^ |   +5 Thank you!It worked fine. The update is a significant improvement.
•  » » » » 4 weeks ago, # ^ |   0 I_love_Hoang_Yen you sure are winning hearts here :D
•  » » » » » 4 weeks ago, # ^ |   0 Thanks =)
•  » » » 3 weeks ago, # ^ |   0 It didn't work out for me. Like it does until page 9 and then when I check my repository, it has done only until some submission of page 8. I start again but it stops at page 9 again. If I start at page 10 instead, it does until some pages(14 or 15) and then again same problem occurs and then again after that. Can u help me out here? :( I would appreciate it
•  » » » » 3 weeks ago, # ^ |   0 Sorry to hear that nmnsharma007 and apologies for getting back to you late. I'm taking a look at what could be the possible issue behind it. Meanwhile, maybe you can use the hack which Mohammad_Yasser proposed of running the script: https://ideone.com/01Yc9C^ It should just be a one-time overhead when you're setting up Harwest for the first time. But I'm on it to identify what could be going wrong for you. Hope that works.
•  » » » » » 3 weeks ago, # ^ | ← Rev. 2 →   0 No problem at all. By the way,where should I put the script?Thanks for the help :) UPD: It worked. Thanks
 » 4 weeks ago, # |   0 It is just me or this tool doesn't work with gym submissions?
•  » » 4 weeks ago, # ^ |   +8 Hey timreizin! Thanks for trying Harwest! You'd be correct that the first iteration of this tool does not support gym submissions or any other submission which is not public :/ The approach was taken to avoid having an extra overhead of requesting and maintaining the user's codeforces password. Though it sure is something that we're looking at including in the next iterations of the tool. In fact, it has already been reported here: https://github.com/nileshsah/harwest-tool/issues/7Will prioritize it in case more people would like to see this feature. Hope that's alright.
•  » » » 4 weeks ago, # ^ |   +5 Thanks for fast reply. Thank you for this tool.
 » 4 weeks ago, # |   +10 Thanks for this amazing tool!Please add support for other platforms as well :)
•  » » 4 weeks ago, # ^ |   +3 Glad you liked it hellodummy! :) We're already working on an integration with AtCoder (all credits to s59_60r). Which other platforms would you like to see on Harwest?
•  » » » 3 weeks ago, # ^ |   +3 How about Codechef?
•  » » » » 3 weeks ago, # ^ |   0
•  » » » 3 weeks ago, # ^ |   0 That's great! Thank you!Other platforms like Codechef, Leetcode :)
 » 4 weeks ago, # |   -10 https://www.stopstalk.com/dashboard something similar to yours.
•  » » 3 weeks ago, # ^ |   0 my bad, I pasted the wrong link earlier. Here is the right link. https://www.stopstalk.com/user/profile/daddy_puff
•  » » 3 weeks ago, # ^ |   +21 Thanks for bringing this up daddy_puff! StopStalk is undoubtedly a wonderful tool to track your activity across different OJs. Huge respect for its creator.Harwest though serves a bit of a different space. It enriches the belongingness of your submissions by allowing you to host them on a platform of your choice rather than being present on a remote system.Harwest fosters the idea of taking immense pride in putting up your solutions on a platform like GitHub since it reflects on the amount of hard work you had put in to succeed early on in your career. As the years go by, you'll realize that all these other tools will fade away but it will only be your Git profile that sticks with you for a while. And I promise you, every time you look back, these contributions will bring back nothing but the good memories of the not so long past: https://github.com/ngthanhtrung23/CompetitiveProgramming
 » 3 weeks ago, # |   0 Thanks for the amazing tool please also consider including codechef, atcoderBug(maybe): After scanning all pages it showing I sovled 319 but my actual solves are 371
•  » » 3 weeks ago, # ^ |   0 Thanks sainad! We'll be launching the support for AtCoder in a matter of few days and will then proceed to add integration for CodeChef. Contributions are always welcomed!The discrepancy that you mentioned might arise from the fact that some of your solutions might not be publicly available, especially gym submissions which Harwest at the moment does not scan. In order to support these private submissions, we'll have to enable login flow on Harwest which we might consider in the future if we have more bandwidth. Hope it is alright for now.
 » 3 weeks ago, # | ← Rev. 2 →   0 Auto comment: topic has been updated by nellex (previous revision, new revision, compare).Support for AtCoder has been added! Please refer to the update section of the blog for further details.
 » 2 weeks ago, # | ← Rev. 2 →   0 GitHubIt seems that I can never get all my submissions (I should have around 600). I've tried harwest codeforces -f multiple times but all of them yield different results, yet none producing all my submissions. Looking at the commmit graph, it seems that harwest ignores a certain period of time, which has happened in all my previous attempts (the actual time is different though). Bug?Edit: When I run something similar to this, I manage to get all my submissions.
•  » » 2 weeks ago, # ^ |   0 Hi skittles1412! Thanks for bringing this up. It sure is quite strange as to why you had to face this problem. I'm mostly guessing that it might be so because some of the requests to fetch the submissions are failing from the server-side. Ideally running a full-scan should have picked up these missed solutions. Hmm, I'll try to reproduce the issue at my end and check if I can locate it. Meanwhile, thanks for sharing a temporary fix.
 » 8 days ago, # | ← Rev. 2 →   0 I believe the atcoder harwest for java submissions does not work
•  » » 7 days ago, # ^ |   +8 Hi bleh0.5! Could you let me know what issue are you facing? Harwest should support Java submissions from AtCoder
•  » » » 7 days ago, # ^ | ← Rev. 3 →   0 Error message: Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\users\bleh0\appdata\local\packages\pythonsoftwarefoundation.python.3.9_qbz5n2kfra8p0\localcache\local-packages\python39\scripts\harwest.exe\__main__.py", line 7, in File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 115, in main args.func(args) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 74, in atcoder process_platform(args, "AtCoder", AtcoderWorkflow) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 90, in process_platform workflow(configs).run(start_page_index=args.start_page, full_scan=full_scan) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 96, in run response.append(self.__add_submission(submission)) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 29, in __add_submission solution_file_path = self.__get_solution_path(submission) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 60, in __get_solution_path lang_ext = config.get_language_extension(submission_lang) File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\utils\config.py", line 49, in get_language_extension raise ValueError( ValueError: ("Please provide correct file extension for the language 'Java8 (OpenJDK 1.8.0)' in", 'C:\\Users\\bleh0\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\harwest\\lib\\resources\\language.json', 'file') I just assumed it had to do with being unable to do java because I recently switched languages
•  » » » » 6 days ago, # ^ |   +8 Thanks for sharing the issue bleh0.5. The necessary file extension has been updated as part of the issue https://github.com/nileshsah/harwest-tool/issues/14Please consider upgrading your Harwest version to 0.3.2 to solve the issue at your end. You can do so by running pip3 install --upgrade harwest Please ensure it says — Successfully installed harwest-0.3.2Do let us know if it still doesn't work for you.
•  » » » » » 6 days ago, # ^ |   0 Ok it works, thanks for the amazing tool!
 » 8 days ago, # |   +5 nellex thank you so much for this awesome tool. Loved it.
 » 6 days ago, # | ← Rev. 2 →   0 return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44b' in position 0: character maps to < undefined >
 » 6 days ago, # |   0 Traceback (most recent call last): File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main return run_code(code, main_globals, None, File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\SHUBHAM\AppData\Local\Programs\Python\Python38\Scripts\harwest.exe__main.py", line 7, in File "c:\users\shubham\appdata\local\programs\python\python38\lib\site-packages\harwest\harwest.py", line 110, in main print("Hey there!", "\U0001F44B", File "c:\users\shubham\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44b' in position 0: character maps to __ __ __ / / / /___ _______ _____ _____/ /_ / /_/ / __ / ___/ | /| / / _ \/ ___/ __// __ / /_/ / / | |/ |/ / ( ) /_ /_/ /_/__,_/_/ |__/|__/___/____/__/==========================================
•  » » 6 days ago, # ^ |   +8 This is a rather interesting case shubham_409, can you run the following commands on your terminal before running Harwest and let me know if it fixes the issue? set PYTHONIOENCODING=utf-8 set PYTHONLEGACYWINDOWSSTDIO=utf-8 
•  » » » 4 days ago, # ^ |   0 Thanks For the quick reply.Still getting same error
•  » » » 4 days ago, # ^ |   0 It worked like a charm using powershell but not by git bash. Thanks for making it. It would be great if it could support other ojs also.
•  » » » » 3 days ago, # ^ |   +3 Glad that you liked it shubham_409. We're looking at on-boarding CodeChef next. Will let you know as soon as it's available :)
•  » » » » » 37 hours ago, # ^ |   0 eagerly waiting for codechef!
 » 2 days ago, # |   0 Can you please tell what's the issue? Spoiler
•  » » 2 days ago, # ^ |   +1 Hey raja_n, thanks for reporting the issue. It's something that we're aware of hence the paragraph in blog about the workaround: As for a workaround for the Windows user, if you want to leverage the automated push feature then specify the remote URL for your repository when asked as https://username:password@github.com/username/repository.git to bypass the login step during the push. Alternatively, you can always leave it empty and do a push from the generated repository by yourself. It was reported on https://github.com/nileshsah/harwest-tool/issues/10 and it turns out that it might get fixed if you update your git version. Can you possibly try that and let me know if that fixes the issue for you as well? Thanks!
•  » » » 2 days ago, # ^ |   0 I have updated git version but now it's showing below error: Error
•  » » » » 2 days ago, # ^ |   +1 Hey raja_n, seems like there are already some files in your CP-Practice repository: https://github.com/Rajan-226/CP-Practice I'd recommend deleting this repository on GitHub and creating a new empty one without any README/Licence files. It should work then.Another alternative is to cd to your local directory of CP Practice and issue a git pull` command to fetch the remote changes onto your local.
•  » » » » » 2 days ago, # ^ |   0 Thanks for replying fast. Yes, it works.Are you planning to add any other platform like codechef, leetcode?
•  » » » » » » 33 hours ago, # ^ |   +1 Adding other platforms would be great!