StatePulse Used AI

Some puppeteer installations on my vm, should be fetching the ai summaries more efficiently now that the libraries have been installed properly. Also met with the managing director at MAD Global Policy + traded emails with senior editor at Tangle regarding StatePulse!

Quick update; mobile friendliness improvement for representative detail pages. Also removed redundant header text for mobile.

Many useful changes made today. Implemented voting power for the interactive dashboard (us house + senate) that shows how much voting power an average person in each state has (compared to Maryland, the base). Changed voted with majority to voted against party in the Voting Stats Summary of each representative. Also enabled users to collapse the sidebar on all screen sizes as well.

User feedback from reddit proved to actually be useful. Implemented on demand detailed ai summaries for congress bills with fetchable pdfs, as well as a compact viewer mode for the policy updates feed. Now users can view more bills with the same screen real estate!

Added learn pages! Now users can learn about the surface level workings of the legislative process directly from the site. Also added a FAQ for common questions. Fixed some ui bugs relating to light/dark mode throughout the site such as the voting record charts

First quality assurance email sent to a random customer using StatePulse! Because of their feedback, I managed to greatly improve the policy tracking filtering for the topic subscriptions. Refactored and deprecated the endpoint to use the unified /api/legislation functionality with appropriate search params

Quick fix enhancing search functionality; now users can search for bills in past voting records and filter yea/nay/pass/fail

Logically, the next step was to add the ability to view rep's voting history! This only works for congress people, but still looks cool... Just a simple mongodb aggregation query continuing with the big backend to get all the vote information for a single rep (yea/nay, vote question, bill topic, etc)

Implemented voting records! A lot of the backend stuff was finished beforehand, but spent a lot of time today and yesterday trying to render the chart correctly using react highcharts. Finally got it to work, and now you can access voting records for congress bills that have them by going to any page!

Switched pdf fetching to puppeteer for dynamic js links that my regular fetcher doesn't catch. Also added tsconfig to my virtual machine. Tried out multiple states with different legislative document links; Texas + Illinois still don't work after several attempts.

Fixed malformed imports for fetchExecutiveOrders.ts and sendPolicyUpdateEmails.ts (for some reason, the env can't read @/module), and updated Included additional checks for existing legislation summaries based on source type and length criteria. Also refined summary prompt in generateOptimizedGeminiSummary function to remove redundant markings for clarity.

Added upsertVotingRecord function to insert or update voting records. Implemented findVotingRecord function to retrieve a voting record by identifier. Spent a lot of time in the library trying to get senate voting records to no avail, as I have to manually scrape them from the web (no api endpoint for them wahh)

Huge codebase refactors mainly through webstorm suggestions. Removed a ton of unused imports and cleaned up some of the most massive files (InteractiveMap.tsx, looking at you!). Also prevented users from activating the representative dialog when map is in full screen to mitigate ui bugs

Some ui map improvements; added loading map progress bar. State lower districts finally doesn't crash on mobile!

Forgot to devlog yesterday and day before yesterday (was sick with fever :(). Anyways, I added a bunch of stuff by improving the enacted bills filter, upserting new congress bills, and changing the .geojson map files to .topojson. Also implemented a full screen version of the map and share buttons!

Numerous more QoL updates! Introduced a feature to filter bills based on if they were enacted to become law by looking at their timeline history and checking if they were signed by a governor/president. Additionally optimized some legislation fetching and map rendering.

Added related bills feature, will need tuning later. Now one to three bills will show that are related to the current bill clicked on. Also enhanced fetching script for congressional bills and added detailed gemini summaries for bills extractable by pdf/full text.

Introduced two new dashboard features; representative + topic heat map! Users can generate heatmaps based on many different metrics such as number of bills sponsored, recent activity, bills relating to immigration, STEM, education, law enforcement, and more. This took way too long to make...

More backend data cleanup; implemented a topic classifier that uses bill title + bill summary to generate topic keywords. Populated topics across all 275k+ bills and updated policy updates feed accordingly to fit new broad topic classifications

Converted a bunch of obselete congress ids from ocd-bill to congress-bill for better consistency. Also created a script to sort bill timelines in case they were malformed by merging duplicates.

Improved notification tracking and added weekly digests! Users can get emails either daily or weekly, with different degrees of insights; additionally, users can also include representatives in the automated newsletter.

Starting to add executive order functionality! Right now StatePulse can track presidential executive orders; still working on adding governor executive orders due to the plethora of websites they are on... Additionally, fixed some more fetching bugs that I encountered with Texas legislation :)

Implemented the voting outcome prediction feature. It takes in all the information about the bill and its sponsors and uses it to determine if it is likely to pass or not. Also implemented rate limiting of this feature so my AI token usage doesn't go completely bonkers...

Some quick qol updates; improved rendering performance on dashboard maps.

Revamped home page to drive user engagement! Added map preview, quick stats, and example bill/rep cards. Also allowed user to follow reps on detailed rep pages.

Implemented gerrymander index based on Polsby-Popper compactness of district geometries; an index between 0 and 1. Perfect circle -> 1, highly irregular shape is close to 0. TODO: improve color scheme

Implemented follow legislators on the frontend, and vastly improved site map and rich metadata specificity. Also converted some client rendered pages to server rendering. Added refresh feed button to fetch legislation API on demand

Fixed session storage issue on reload, and also the view all sponsored bills bug that arose after yesterday's data migration! Corrected the id use for the representative and pushed to next navigation link

Lots of QoL improvements in this update! Implemented sponsor linking in legislative pages, so users can now check the stats of representatives who've sponsored a particular bill. Additionally, for congress people at least, there is much more information after fetching more detailed stats through the congress API :)) Also fixed various bugs that arose from data migration

Added daily automated topic tracking email newsletters! Users can choose which topics they want in their daily inbox. Used SMTP/Brevo.

New feature time! Incorporated a toggle that enables users to show which party represents which district: congressional, state upper, or state lower. Had to reexport all of the .geojsons cause they didn't include the right property fip/geocode matching the first time.

More refactoring internally, also starting to implement backend that enables users to follow legislators!

Minimal feature changes; just a lot of codebase refactoring!

Many QoL improvements in this devlog! Fixed some dark mode theming issues for the map popups/key. Replaced Leaflet rendering engine with MapLibre GL, which now renders all of the legislative districts much more quickly than before. No lag! Added more bills and merged duplicate representatives as well.

Added chamber makeup into dashboard stats; users can now look at which party controls which chamber in different states, including numbers and percentages!

Added previous terms for Congress people in the representatives detail page. Also refined tables/buttons to improve consistency and dashboard stats to make circle size proportional to number of bills

Users can now click anywhere on the map and drag the location icon to get the representatives that represent those coordinates!

Added .geojson files client side for faster district boundary loading.

Federal and state legislative boundaries are here! Imported them into mongodb from U.S. Census. Implemented find your local reps, which get the relevant districts using lat + long. Also added all district boundary visualizations in the dashboard, and color coding representative cards for ux :)

Lots of new features in this devlog; implemented fuzzy search in policy updates feed using fuse.js. Added the field jurisdictionName: United States Congress for all Congress bills, so fetching bills from Congress should be nearly instant now. Also created some scripts to clean up some unwanted summaries.

Added legislation existence check so I don't call unnecessary gemini summaries from the API

Implemented time in office, added direct legislation filtered by representative in policy updates feed, and fixed clear filter bug not reloading feed state. Improved UI and UX for representative details with animations and better buttons!

Finished adding representative collection in mongodb database. Normalized all Congress representatives and senators into correct data types; now users can click on any representative to see the bills that they have sponsored! Obviously still a WIP, as some time dates are curiously NaN...

Changed source fetching script to have an Illinois exception (due to their stupid legislative page format)

Lots of new features in this one! Added distinct representative page for lookup and filtering, donation buttons, and a parallax animation on the home page to boost engagement. Implemented civics endpoint to remove conflicts with representative endpoint, and switched back to Google AI Studio after looking at the costs :D

Continued changing the AI Summaries text extraction from the different pdfs provided by legislature websites! Switched from Google AI Studio to Microsoft Azure AI for cost purposes. Text extraction from pdfs and source links now works better than ever...

Implemented SMTP email senders in tracking policies! Also fixed some of the backend API for handling push notifications, and added a new email template. Settled on Brevo for SMTP after hours of searching for alternatives to gmail. Unrelated, transferred DNS from Namecheap to Cloudflare :)

Lots of new stuff in this one! Spent a while refactoring the PostsFeed and PostCard components, which enabled me to add liking, more sorting (new, hot, top, and old), and display exact times. Also fixed some CRUD issues that came up through previous updates.

Updated the fetching script to include more robust Gemini Summary handling by feeding it better input text. This bug has been bashing me since the existence of this project...

Users can now search posts and filter them by tags!

Fixed CRUD operations on detailed individual community posts. Users can also click into posts from viewing other people's profiles.

Implemented some functionality into viewing individual community posts on a separate page. CRUD operations will be fixed soon.

More animations added! Also created a script that merged duplicate Congress bills containing same source links but having different identifiers.

Reformatted a ton of code. Added animations to the entire website, reduced loading times!

Fixed dashboard data inconsistencies in internal API

Major internal codebase changes, refactored a ton of types. Also fixed fetching representatives bug, where states with similar acronyms would fetch cached reps of different states (i.e. CA and AL)

Fixed incorrect Congressional session aggregation for dashboard. Implemented more robust solution for querying most recent Congress session; floor((x - 1789 / 2) + 1, where x is the current year

Fixed bugs regarding searching for Congress bills in the policy updates feed. Also enabled aggregation for U.S. Congress in dashboard.

Added state-specific dashboard statistics! Users can now look at popular topics, most active representatives, and recent bills at a glance.

Users can now view a dashboard to see overview statistics of all state legislation in the U.S. More features will be implemented soon!

Legislation fetching script now handles two APIs: OpenStates and U.S. Congress.

Bug fixes: fixed incorrect pagination state on new state search, fixed multiword queries for RepresentativesFinder.tsx (states like New York, New Jersey, New Mexico were calling directly from the OpenStates API instead of from mongodb cache); Enhancements: added custom tags for policy search, enabled users to view details on embedded legislation in community posts

Users can now add comments to posts, with fully implemented CRUD operations. Comments are nested akin to Instagram/Youtube/X.

Users can now create posts and link legislation to express their views! Also enhanced mobile UI and deployed a production instance of Clerk Auth. Core features are essentially complete (except for the occasional bug :)).

Added generate message to legislator. Now users can contact with their state rep/senator and ask questions pertaining to their bills!

Added civics tool improved mobile formatting (still need to iron out more bugs), added bookmark functionality

Added further improvements in policy tracker front/back end, barebone functionality

Changed timeline representation format

Started implementing page protection, used mistral from ollama to generate better summaries of policies, pushed to mongo

Added ollama scripts to generate summaries of legislation locally + push to mongo db database, continued populating historical legislation, fixed infinite scroll + state preservation on policy update feed

Added a better script to fetch all legislation data from all fifty states from 2017 to present

Improved pagination, switched localStorage caching to sessionStorage

Added more historical legislation

Added night theme toggle

Added home page