-
-
Notifications
You must be signed in to change notification settings - Fork 4.8k
Allow Parse Job to be launched with more than 16Mb of data in params #8341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Thanks for opening this issue!
|
In that context, the params stored cannot have any operational value, they could have at most analytical or troubleshooting value. In addition, storing operational data in an internal collection is not supposed to happen and we should probably implement a guideline for reviewers about that. Operational data may be sensitive, require data protection considerations, auditing, or other things that are problematic for internal collections. My suggestion is to remove param logging. If params are currently used by Parse Server, then we could think about finding a replacement concept. Any logging of params can be implemented fine-grained and according to use case by the developer in the job script itself. |
@mtrezza So if I understand you correctly, you think we should remove the |
I would say so, unless there is a good reason why they are logged. Did you look at the code to see whether they are used anywhere? |
The only good reason to store the |
Good to know that they aren't used. So I'd suggest to remove then completely. |
If I understood the discussion correctly, the solution would be to keep this file in a separate location like a ParseFile, for example. So it would only be necessary to pass an objectId as a Job parameter. Although today the parameters are not shown on the Job Status page, I would prefer not to remove them, as simply removing them does not improve privacy or security at all. I've also thought about the possibility of making a retry mechanism, but I never had the time.
|
The params are only removed from the collection, they are still passed through the request to the Parse.Job function so no worries about that. |
@danielsanfr did that answer address your concerns? |
🎉 This change has been released in version 8.2.1-alpha.2 |
New Feature / Enhancement Checklist
Current Limitation
When running a Parse Job, the params is saved in the _JobStatus collection by default in MongoDB thus limiting the size of the params to the MongoDB BSON limit of 16Mb : https://www.mongodb.com/docs/manual/reference/limits/#:~:text=The%20maximum%20BSON%20document%20size,MongoDB%20provides%20the%20GridFS%20API.
Feature / Enhancement Description
We would need to modify this code :
parse-server/src/Routers/FunctionsRouter.js
Line 73 in 4c29d4d
Two ways of doing it :
Example Use Case
For example I want to launch a job with 32Mb of data to analyse, I will launch the job but explicitly telling Parse to not save the params in database.
Alternatives / Workarounds
No easy workaround
3rd Party References
POSTing more than 16Mb of data is quite common everywhere :)
The text was updated successfully, but these errors were encountered: