nellex's blog

By nellex, history, 4 weeks ago, In English

Hi beautiful people of Codeforces! Hope you'd all be doing fine and keeping safe.

I'd like to start by mentioning that it's been a while since I graduated from university ever since which I haven't completely been able to indulge myself in competitive coding in the same way I did back in my university. But I sure haven't forgotten those days. I had some time off from work recently and decided to spend my time giving back to the community from where I have learned so much.

In the thought of so, I've built this tool which I like to call Harwest (a play on the word Harvest since the package name wasn't available anymore :'D) It's one of those tools which I would have loved to have during the time when I used to practice rigorously on this platform.

Harwest allows you to manage all the submissions that you've made on Codeforces, or on other OJs (WIP) into one single Git repository which you can then push to your personal Github / BitBucket / Gitlab or any other SCM platform of your choice.

Harwest completely automates this entire process with minimal effort from your side of answering just a few basic questions, and even goes to the limit of pushing the Git repository to the remote (Github/BitBucket..) for you!

Here's a sample repository built solely using Harwest: https://github.com/nileshsah/harwest-sample

The project directory structure is as follows:

```
nellex@HQ:~/accepted$ tree
.
├── codeforces
│   ├── <contest-id>
│   │   └── <problem-level>
│   └── 592
│       └── A.cpp
├── README.md
├── submissions.json
└── .git
```

Harwest creates a local git repository for you, scrapes all your submissions directly from Codeforces, maintains them in a nicely organized way, and commits them to the repository with the actual date of your submission!

Now what the last statement enables us to do is to maintain a complete contribution graph on GitHub (or others) accurate to the time when you actually made the submission and not when you made the commit. Yes! It means complete tracking of your submissions on a day by day basis. Keep track of the number of days you've been lazy for, the number of problems you solved on a given day, and maintain a complete streak of daily practice! Heck, let this streak be your new motivation to practice continuously to become a better version of yourself every single day

I've detailed the entire process of configuring Harwest on your local computer with an example in https://github.com/nileshsah/harwest-tool#installation Let me know if something needs more clarification.

It's built using Python so you would require PIP with Python version 3.5 or above to install the package which you might have to set up in case you haven't used it before. It's a great package manager which I'd recommend learning a bit about nonetheless since it'll come in handy in the future.

Along with PIP, you'd also need git on your computer which would come pre-installed if you're using Linux or MacOS though might would have to work your way up in case you're using windows. I'd also strongly encourage you to learn about Git since it's something that is heavily used in the software industry and early exposure to it would sure make your life a lot easier in the future. There are numerous documents on the internet which can help you get started on Git, one of them is https://www.freecodecamp.org/news/an-introduction-to-git-for-absolute-beginners-86fa1d32ff71/ Though For operating Harwest, you'd require minimal to no experience with Git.

The first time Harwest runs, it'll scrape all your submissions starting from the first page till the last which might be a bit time-consuming if you had made a lot of submissions. If you don't want to parse the results till the very end, you can stop processing by hitting <Ctrl> + C. The next runs of Harwest will no more try to scrape the results from the old pages and instead will push your new changes directly. You can then make use of the harwest codeforces --start-page X feature to restart scraping from the page where you left off.

I haven't got a chance to extensively test Harwest much and hence the term [Beta]. It seems to be working fine on MacOS and Ubuntu but I guess the auto push feature is causing a bit of an issue on Windows especially if you're using GitHub for Windows integration Do keep me posted if you face any other issues and I'd be happy to fix it at once.

As for a workaround for the Windows user, if you want to leverage the automated push feature then specify the remote URL for your repository when asked as https://username:password@github.com/username/repository.git to bypass the login step during the push. Alternatively, you can always leave it empty and do a push from the generated repository by yourself.

Nonetheless, I had a fun time building this tool and I really hope it would be helpful for you. Contributions to the project are more than welcome!

TL;DR

Automate the management of your submissions on GitHub with https://github.com/nileshsah/harwest-tool to get something like https://github.com/nileshsah/harwest-sample


Update <2020-12-27>: Support for AtCoder is now available!

Harwest now supports AtCoder! Massive thanks to s59_60r for leading the effort of finding the ideal way to integrate with the platform. Not to forget the crucial contributions from I_love_Hoang_Yen, YouKn0wWho, and others for fixing some major performance issues and making Harwest more stable. Much appreciated.

The sample submissions repository https://github.com/nileshsah/harwest-sample has been updated to reflect what new changes this update brings. In order to switch to this release, please upgrade Harwest by running:

pip install --upgrade harwest

Ensure that you're on version harwest-0.3.1, The new release is backward compatible and will not break any of your existing harvested submissions. After switching to the new version, you can then harvest submissions from AtCoder by running

harwest atcoder

In addition to the new integration with AtCoder, we have introduced --full-scan capability on platforms to avoid the pain of dealing with re-starting harwest from the page where it stopped processing due to an error. It can be triggered as:

harwest codeforces --full-scan
OR
harwest atcoder --full-scan

Please do let me know if you face any difficulties in the process, would be happy to help at once.

Happy Harwesting! :)

 
 
 
 
  • Vote: I like it
  • +270
  • Vote: I do not like it

»
4 weeks ago, # |
  Vote: I like it 0 Vote: I do not like it

Auto comment: topic has been updated by nellex (previous revision, new revision, compare).

»
4 weeks ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

I installed Harwest but when I try to run the command.

It says harwest: command not found.

I’m using Ubuntu.

  • »
    »
    4 weeks ago, # ^ |
    Rev. 3   Vote: I like it +8 Vote: I do not like it

    Hey sam571128! Thanks for trying. I'm guessing it might be so because the directory under which pip installed the package may not be present in your PATH. Could you try running the command pip3 uninstall harwest

    ```
    nellex@HQ:~$ pip3 uninstall harwest
    Found existing installation: harwest 0.2.0
    Uninstalling harwest-0.2.0:
      Would remove:
        /home/nellex/.local/bin/harwest
        /home/nellex/.local/lib/python3.5/site-packages/harwest-0.2.0.dist-info/*
        /home/nellex/.local/lib/python3.5/site-packages/harwest/*
    Proceed (y/n)? n
    ```
    

    > /home/nellex/.local/bin/harwest

    The path /home/nellex/.local/bin is something you'd like to register under $PATH as export PATH=/home/nellex/.local/bin:$PATH or you can execute the command directly as /home/nellex/.local/bin/harwest

    Let me know if that works.

    • »
      »
      »
      4 weeks ago, # ^ |
        Vote: I like it +5 Vote: I do not like it

      Sorry for replying so late. But yea, it worked.

»
4 weeks ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

This is such a good idea! unfortunately I'm getting an error when running harwest codeforces (and putting in my handle):

equest.py", line 1352, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>
  • »
    »
    4 weeks ago, # ^ |
    Rev. 6   Vote: I like it 0 Vote: I do not like it

    Thanks a ton AkshajK I've updated the code to skip SSL verification. Can you try installing the new version by running pip3 install harwest==0.2.9 Let me know if the problem persists.

»
4 weeks ago, # |
Rev. 3   Vote: I like it +19 Vote: I do not like it

Would like to extend my heartiest thanks to YouKn0wWho for helping me out in fixing some of the major encoding issues in Harwest on the windows platform. In case you were a windows user and have already tried Harwest then you might notice ^M characters in the generated submission files. If so, then I'd recommend deleting the locally created repository and starting over with harwest --init after updating to the latest version by running pip3 install harwest==0.2.9 Apologies for the inconvenience.

As for YouKn0wWho, the amount of hard work you've put in this year itself is simply remarkable! https://github.com/ShahjalalShohag/ProblemSolving

»
4 weeks ago, # |
  Vote: I like it 0 Vote: I do not like it

I have installed it and pushed my submissions to github. Now if I make more submissions, how do I push them again so that it only takes the submissions which are not pushed yet?

  • »
    »
    4 weeks ago, # ^ |
    Rev. 3   Vote: I like it 0 Vote: I do not like it

    Hey anshu_2761! Thanks for giving Harwest a try! Harwest keeps track of your past submissions and fetches only the ones which are new. Whenever you'd like to sync the submissions in your GitHub repository with that on Codeforces, just run harwest codeforces and it'll do the job for you. Will also push automatically if you have set it up with auto-push else you can manually cd into the local submissions directory and issue a git push origin master yourself.

    I believe this was your question right? Also, I'd recommend running a pip3 install harwest==0.2.9 again since I pushed in some new set of fixes which would be good to pull.

»
4 weeks ago, # |
  Vote: I like it +5 Vote: I do not like it

Nice tool!

However, the process was stopping a lot before finishing all pages, and I had to use the commnad harwest codeforces --start-page x multiple times, then I wrote this python snippet to do the job.

import subprocess
for i in range(300):
  subprocess.call(['harwest', 'codeforces', '--start-page', str(i)])
  • »
    »
    4 weeks ago, # ^ |
    Rev. 2   Vote: I like it +13 Vote: I do not like it

    Thanks Mohammad_Yasser! Nice hack I'd say! I would like to apologize that I didn't stress-test the tool enough to check its breaking limits. I_love_Hoang_Yen just helped me out in introducing a performance update (PR#3) which should address your issue a bit. Updating to the latest version of Harwest should help. Could you try executing the commands:

    ```
    pip uninstall harwest
    pip install harwest
    ```
    

    If it doesn't fetch the latest version of 0.2.10 then maybe you can do a force install using pip install harwest==0.2.10

    Let me know if that helps and thanks for trying Harwest out :)

    • »
      »
      »
      4 weeks ago, # ^ |
        Vote: I like it +5 Vote: I do not like it

      Thank you!

      It worked fine. The update is a significant improvement.

    • »
      »
      »
      4 weeks ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      It didn't work out for me. Like it does until page 9 and then when I check my repository, it has done only until some submission of page 8. I start again but it stops at page 9 again. If I start at page 10 instead, it does until some pages(14 or 15) and then again same problem occurs and then again after that. Can u help me out here? :( I would appreciate it

      • »
        »
        »
        »
        4 weeks ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Sorry to hear that nmnsharma007 and apologies for getting back to you late. I'm taking a look at what could be the possible issue behind it. Meanwhile, maybe you can use the hack which Mohammad_Yasser proposed of running the script: https://ideone.com/01Yc9C

        ^ It should just be a one-time overhead when you're setting up Harwest for the first time. But I'm on it to identify what could be going wrong for you. Hope that works.

        • »
          »
          »
          »
          »
          4 weeks ago, # ^ |
          Rev. 2   Vote: I like it 0 Vote: I do not like it

          No problem at all. By the way,where should I put the script?Thanks for the help :) UPD: It worked. Thanks

»
4 weeks ago, # |
  Vote: I like it 0 Vote: I do not like it

It is just me or this tool doesn't work with gym submissions?

  • »
    »
    4 weeks ago, # ^ |
      Vote: I like it +8 Vote: I do not like it

    Hey timreizin! Thanks for trying Harwest!

    You'd be correct that the first iteration of this tool does not support gym submissions or any other submission which is not public :/ The approach was taken to avoid having an extra overhead of requesting and maintaining the user's codeforces password. Though it sure is something that we're looking at including in the next iterations of the tool. In fact, it has already been reported here: https://github.com/nileshsah/harwest-tool/issues/7

    Will prioritize it in case more people would like to see this feature. Hope that's alright.

    • »
      »
      »
      4 weeks ago, # ^ |
        Vote: I like it +5 Vote: I do not like it

      Thanks for fast reply. Thank you for this tool.

»
4 weeks ago, # |
  Vote: I like it +10 Vote: I do not like it

Thanks for this amazing tool!

Please add support for other platforms as well :)

»
4 weeks ago, # |
  Vote: I like it -10 Vote: I do not like it

https://www.stopstalk.com/dashboard something similar to yours.

  • »
    »
    4 weeks ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    my bad, I pasted the wrong link earlier. Here is the right link. https://www.stopstalk.com/user/profile/daddy_puff

  • »
    »
    4 weeks ago, # ^ |
      Vote: I like it +21 Vote: I do not like it

    Thanks for bringing this up daddy_puff! StopStalk is undoubtedly a wonderful tool to track your activity across different OJs. Huge respect for its creator.

    Harwest though serves a bit of a different space. It enriches the belongingness of your submissions by allowing you to host them on a platform of your choice rather than being present on a remote system.

    Harwest fosters the idea of taking immense pride in putting up your solutions on a platform like GitHub since it reflects on the amount of hard work you had put in to succeed early on in your career. As the years go by, you'll realize that all these other tools will fade away but it will only be your Git profile that sticks with you for a while. And I promise you, every time you look back, these contributions will bring back nothing but the good memories of the not so long past: https://github.com/ngthanhtrung23/CompetitiveProgramming

»
4 weeks ago, # |
  Vote: I like it 0 Vote: I do not like it

Thanks for the amazing tool please also consider including codechef, atcoder

Bug(maybe): After scanning all pages it showing I sovled 319 but my actual solves are 371

  • »
    »
    4 weeks ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Thanks sainad! We'll be launching the support for AtCoder in a matter of few days and will then proceed to add integration for CodeChef. Contributions are always welcomed!

    The discrepancy that you mentioned might arise from the fact that some of your solutions might not be publicly available, especially gym submissions which Harwest at the moment does not scan. In order to support these private submissions, we'll have to enable login flow on Harwest which we might consider in the future if we have more bandwidth. Hope it is alright for now.

»
3 weeks ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

Auto comment: topic has been updated by nellex (previous revision, new revision, compare).

Support for AtCoder has been added! Please refer to the update section of the blog for further details.

»
3 weeks ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

GitHub
It seems that I can never get all my submissions (I should have around 600). I've tried harwest codeforces -f multiple times but all of them yield different results, yet none producing all my submissions. Looking at the commmit graph, it seems that harwest ignores a certain period of time, which has happened in all my previous attempts (the actual time is different though). Bug?

Edit: When I run something similar to this, I manage to get all my submissions.

  • »
    »
    3 weeks ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Hi skittles1412! Thanks for bringing this up. It sure is quite strange as to why you had to face this problem. I'm mostly guessing that it might be so because some of the requests to fetch the submissions are failing from the server-side. Ideally running a full-scan should have picked up these missed solutions. Hmm, I'll try to reproduce the issue at my end and check if I can locate it. Meanwhile, thanks for sharing a temporary fix.

»
11 days ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

I believe the atcoder harwest for java submissions does not work

  • »
    »
    11 days ago, # ^ |
      Vote: I like it +8 Vote: I do not like it

    Hi bleh0.5! Could you let me know what issue are you facing? Harwest should support Java submissions from AtCoder

    • »
      »
      »
      10 days ago, # ^ |
      Rev. 3   Vote: I like it 0 Vote: I do not like it

      Error message:

      Traceback (most recent call last):
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
          return _run_code(code, main_globals, None,
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
          exec(code, run_globals)
        File "C:\users\bleh0\appdata\local\packages\pythonsoftwarefoundation.python.3.9_qbz5n2kfra8p0\localcache\local-packages\python39\scripts\harwest.exe\__main__.py", line 7, in <module>
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 115, in main
          args.func(args)
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 74, in atcoder
          process_platform(args, "AtCoder", AtcoderWorkflow)
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\harwest.py", line 90, in process_platform
          workflow(configs).run(start_page_index=args.start_page, full_scan=full_scan)
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 96, in run
          response.append(self.__add_submission(submission))
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 29, in __add_submission
          solution_file_path = self.__get_solution_path(submission)
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\abstractworkflow.py", line 60, in __get_solution_path
          lang_ext = config.get_language_extension(submission_lang)
        File "C:\Users\bleh0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\harwest\lib\utils\config.py", line 49, in get_language_extension
          raise ValueError(
      ValueError: ("Please provide correct file extension for the language 'Java8 (OpenJDK 1.8.0)' in", 'C:\\Users\\bleh0\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\harwest\\lib\\resources\\language.json', 'file')
      

      I just assumed it had to do with being unable to do java because I recently switched languages

      • »
        »
        »
        »
        9 days ago, # ^ |
          Vote: I like it +8 Vote: I do not like it

        Thanks for sharing the issue bleh0.5. The necessary file extension has been updated as part of the issue https://github.com/nileshsah/harwest-tool/issues/14

        Please consider upgrading your Harwest version to 0.3.2 to solve the issue at your end. You can do so by running pip3 install --upgrade harwest Please ensure it says — Successfully installed harwest-0.3.2

        Do let us know if it still doesn't work for you.

        • »
          »
          »
          »
          »
          9 days ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          Ok it works, thanks for the amazing tool!

»
11 days ago, # |
  Vote: I like it +5 Vote: I do not like it

nellex thank you so much for this awesome tool. Loved it.

»
9 days ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44b' in position 0: character maps to < undefined >

»
9 days ago, # |
  Vote: I like it 0 Vote: I do not like it

Traceback (most recent call last): File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main return run_code(code, main_globals, None, File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\SHUBHAM\AppData\Local\Programs\Python\Python38\Scripts\harwest.exe__main.py", line 7, in File "c:\users\shubham\appdata\local\programs\python\python38\lib\site-packages\harwest\harwest.py", line 110, in main print("Hey there!", "\U0001F44B", File "c:\users\shubham\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44b' in position 0: character maps to

__  __                              __
 / / / /___ _______      _____  _____/ /_
/ /_/ / __ `/ ___/ | /| / / _ \/ ___/ __/

/ __ / /_/ / / | |/ |/ / ( ) /_ /_/ /_/__,_/_/ |__/|__/___/____/__/

==========================================

  • »
    »
    9 days ago, # ^ |
      Vote: I like it +8 Vote: I do not like it

    This is a rather interesting case shubham_409, can you run the following commands on your terminal before running Harwest and let me know if it fixes the issue?

    set PYTHONIOENCODING=utf-8
    set PYTHONLEGACYWINDOWSSTDIO=utf-8
    
    • »
      »
      »
      7 days ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Thanks For the quick reply.Still getting same error

    • »
      »
      »
      7 days ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      It worked like a charm using powershell but not by git bash. Thanks for making it. It would be great if it could support other ojs also.

      • »
        »
        »
        »
        6 days ago, # ^ |
          Vote: I like it +3 Vote: I do not like it

        Glad that you liked it shubham_409. We're looking at on-boarding CodeChef next. Will let you know as soon as it's available :)

»
6 days ago, # |
  Vote: I like it 0 Vote: I do not like it

Can you please tell what's the issue?

Spoiler
  • »
    »
    6 days ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    Hey raja_n, thanks for reporting the issue. It's something that we're aware of hence the paragraph in blog about the workaround:


    As for a workaround for the Windows user, if you want to leverage the automated push feature then specify the remote URL for your repository when asked as https://username:password@github.com/username/repository.git to bypass the login step during the push. Alternatively, you can always leave it empty and do a push from the generated repository by yourself.

    It was reported on https://github.com/nileshsah/harwest-tool/issues/10 and it turns out that it might get fixed if you update your git version. Can you possibly try that and let me know if that fixes the issue for you as well? Thanks!

    • »
      »
      »
      5 days ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      I have updated git version but now it's showing below error:

      Error
      • »
        »
        »
        »
        5 days ago, # ^ |
          Vote: I like it +1 Vote: I do not like it

        Hey raja_n, seems like there are already some files in your CP-Practice repository: https://github.com/Rajan-226/CP-Practice I'd recommend deleting this repository on GitHub and creating a new empty one without any README/Licence files. It should work then.

        Another alternative is to cd to your local directory of CP Practice and issue a git pull command to fetch the remote changes onto your local.

        • »
          »
          »
          »
          »
          5 days ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          Thanks for replying fast. Yes, it works.

          Are you planning to add any other platform like codechef, leetcode?

          • »
            »
            »
            »
            »
            »
            5 days ago, # ^ |
              Vote: I like it +1 Vote: I do not like it

            Adding other platforms would be great!