Skip to content

Allow Parse Job to be launched with more than 16Mb of data in params #8341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
codeKonami opened this issue Nov 30, 2022 · 10 comments · Fixed by #8343
Closed
3 tasks done

Allow Parse Job to be launched with more than 16Mb of data in params #8341

codeKonami opened this issue Nov 30, 2022 · 10 comments · Fixed by #8343
Labels
state:released-alpha Released as alpha version type:feature New feature or improvement of existing feature

Comments

@codeKonami
Copy link
Contributor

New Feature / Enhancement Checklist

Current Limitation

When running a Parse Job, the params is saved in the _JobStatus collection by default in MongoDB thus limiting the size of the params to the MongoDB BSON limit of 16Mb : https://www.mongodb.com/docs/manual/reference/limits/#:~:text=The%20maximum%20BSON%20document%20size,MongoDB%20provides%20the%20GridFS%20API.

Feature / Enhancement Description

We would need to modify this code :

return jobHandler.setRunning(jobName, params).then(jobStatus => {

Two ways of doing it :

  • We could pass a context while launching the Parse Job telling that we should not save the params.
  • We could measure the size of the params and therefore truncate the params inside the JobStatus document.

Example Use Case

For example I want to launch a job with 32Mb of data to analyse, I will launch the job but explicitly telling Parse to not save the params in database.

Alternatives / Workarounds

No easy workaround

3rd Party References

POSTing more than 16Mb of data is quite common everywhere :)

@parse-github-assistant
Copy link

Thanks for opening this issue!

  • 🎉 We are excited about your ideas for improvement!

@mtrezza
Copy link
Member

mtrezza commented Nov 30, 2022

_JobStatus is an internal collection and therefore not supposed to be used for anything else than supporting features of Parse Server. Any change to that collection would not constitute a breaking change, so one can not rely on it, what is logged, how it's logged, or whether anything is logged at all.

In that context, the params stored cannot have any operational value, they could have at most analytical or troubleshooting value. In addition, storing operational data in an internal collection is not supposed to happen and we should probably implement a guideline for reviewers about that. Operational data may be sensitive, require data protection considerations, auditing, or other things that are problematic for internal collections.

My suggestion is to remove param logging. If params are currently used by Parse Server, then we could think about finding a replacement concept. Any logging of params can be implemented fine-grained and according to use case by the developer in the job script itself.

@mtrezza mtrezza added the type:feature New feature or improvement of existing feature label Nov 30, 2022
@codeKonami
Copy link
Contributor Author

@mtrezza So if I understand you correctly, you think we should remove the params altogether?

@mtrezza
Copy link
Member

mtrezza commented Dec 2, 2022

I would say so, unless there is a good reason why they are logged. Did you look at the code to see whether they are used anywhere?

@codeKonami
Copy link
Contributor Author

The only good reason to store the params in the JobStatus document would be to have a retry mechanism of some sort. But as of today these params does not seem to be used anywhere.

@mtrezza
Copy link
Member

mtrezza commented Dec 2, 2022

Good to know that they aren't used. So I'd suggest to remove then completely.

@danielsanfr
Copy link

danielsanfr commented Dec 8, 2022

If I understood the discussion correctly, the solution would be to keep this file in a separate location like a ParseFile, for example. So it would only be necessary to pass an objectId as a Job parameter.

Although today the parameters are not shown on the Job Status page, I would prefer not to remove them, as simply removing them does not improve privacy or security at all.

I've also thought about the possibility of making a retry mechanism, but I never had the time.
So, if you still want to remove something... Please, just remove it from the collection, because I have several Jobs using parameters and I believe that there are many more users who also use them.

Please disregard the comment, if I understood something wrong or if I didn't express myself well, because English is not my main language.

@codeKonami
Copy link
Contributor Author

If I understood the discussion correctly, the solution would be to keep this file in a separate location like a ParseFile, for example. So it would only be necessary to pass an objectId as a Job parameter.

Although today the parameters are not shown on the Job Status page, I would prefer not to remove them, as simply removing them does not improve privacy or security at all.

I've also thought about the possibility of making a retry mechanism, but I never had the time. So, if you still want to remove something... Please, just remove it from the collection, because I have several Jobs using parameters and I believe that there are many more users who also use them.

Please disregard the comment, if I understood something wrong or if I didn't express myself well, because English is not my main language.

The params are only removed from the collection, they are still passed through the request to the Parse.Job function so no worries about that.

@mtrezza
Copy link
Member

mtrezza commented Dec 8, 2022

@danielsanfr did that answer address your concerns?

codeKonami added a commit to codeKonami/parse-server that referenced this issue May 8, 2025
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 8.2.1-alpha.2

@parseplatformorg parseplatformorg added the state:released-alpha Released as alpha version label May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state:released-alpha Released as alpha version type:feature New feature or improvement of existing feature
Projects
None yet
4 participants