Develop to Main. Sprint 2 #54
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Deleted non-relevant classes for web-scraping objects/methods.
Added selenium as a new dependency.
Added class to scrape confirmed organizations from InnsamlingsKontrollen.
Added class to store values from scraped organizations.
Changed some of the logic in updateData() to prepare for format for writing to csv file.
Added opencsv as a dependency.
Changed the logic for updateData to be more robust (old method didn't get all data reliably). Put method in a try - finally statement to ensure headless chrome browser reliably shuts down after completion/if error. Added a method to write gathered data to a csv file. Made the getter method return an immutable list.
Added method deleteCSV that deletes the csv if it exists. Added method to writeToCSV for cleaner generation of csv. Also added exceptions to handle missing data and failure to delete CSV file.
Changed parameters to private immutable parameters.
Quick test class to check that the Organization object is created successfully with correct parameters.
Added GSON as a Maven dependency.
Added a class for scraping the IK API and parsing the JSON values, pushing them to a APICharityData object for easy migrating to a database later.
Added testclass for the scraper with some test methods.
Added mockito as a dependency to make testing of http requests easier.
Replaced the connection/get request logic to make it easier to mocktest with mockito.
Changed test to use mocked http-responses using mockito for more reliable testing.
Added test for constructor/getters for ApiCharityData.
Added mysql-connector-j as a dependency.
Added a streams filter to remove "obs" status values in the JSON data from the API, due to non-unique org_number values. Org_number is used as primary key, so this caused issues with database.
Working manager class for handling creation, insertion, and deletion of tables and records. Currently, supports creating charities table, updating the table with values from the IK API, and doing a parity check, removing values that doesn't exist in the IK API.
Added javadoc descriptions, throws an exception if org_number is invalid, and automatically normalizes the value of org_number.
Added tests for new logic regarding org_number.
Added JavaDocs.
Added a test to check that JSON entries with status:'obs' gets ignored.
Added JavaDocs and printStackTrace to createCharitiesTable method.
Made sql statements into transaction statements for increased stability. Converted "temp" table into a fully temporary table. Reused a single String variable for sql queries instead of one variable per statement.
Added H2 as a dependency, an in-memory mock database, useful for testing DatabaseManager logic.
Added a second constructor to make JUnit testing of the class feasible. Second constructor should not be used to create object in production code.
Added some coverage tests to check if it functions as intended. Does not currently have 100% coverage.
Removed IF NOT EXISTS for temporary table as it gave unintended results in unit tests.
…e on a later date
…nto feature/40-mvp-funksjonality-across-sites
Sign in
to join this conversation on GitHub.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Good job everyone