Running a Spark Submit Job Locally

For testing you may need to run a spark submit job lifted from a convert to test something like Humpty

Steps

Update local bettrdata instance
Start local instace
Set up address standardization in instance
Set up convert type to use address standardization
Load a dataset to do the standardization
Set up the convert with the data file
Run a production convert to get anchor running
Check the log for the second spark-submit, which is the Humpty part

To Develop you do these steps

Open the transform-io project in intellij or vscode
Make the code changes as needed
Update and move updated jar into docker


docker_build.sh
copy-local.sh

Restart the local backend to test:

./stop.sh
./start.sh

Check the logs in spark-events. Add in print or log statements for more details