1. We have developed a microservice only for posting gossips using node js and express and multiple other technologies and libraries.
2. We have created a Rest API endpoint using node js and express for receiving post gossip requests.
3. We have adopted the separation of concerns pattern in our program, wherein, we have the route handlers and controllers, which belong to the web layer, and then we have the business logic layer, which contains our services, and then, at last, we have the DAL layer which is the only part of program allowed to communicate with the database.
4. Then, when the client sends a request to our backend, first the request needs to go through the middleware layer, and only then it can reach our controllers.
5. In our middleware layer, we have the three middleware's setup, one is the authentication middleware and the other is the middleware library called multer which allows us to work with multipart form data, and the last middleware is the JSON schema validation middleware.
6. The request that we receive from the client must contain the authentication key in their header, if there is no AUTH key or if it contains the wrong AUTH key then that request will be rejected with the 401 status code and with an error response.
7. If the request has the correct AUTH key, then it needs to go through the multer middleware, which processes our client's request which is in multipart/form-data and helps us to work with it. It is primarily used to process the images coming from our client-side.
8. Then the request needs to go through the JSON schema validation middleware, this is set up to make sure that the request that we are receiving from the client-side has all the required data and the proper type of data, to achieve this we are using a library called ajv, with the help of this library we define the schema of the incoming request beforehand and once we receive the request, we validate it against the prepared schema, if the request doesn't abide by the schema then we send a 400 bad request status code along with the details of the improper schema.
9. Once the request goes through all these middleware's only then it touches our controller, which communicates with our business logic layer, which has the services.
10. We have different services, some of those are save gossip, save image, delete gossip, delete image, malicious URL detection, image moderation, and bad words filter, all these services are pretty much self-explanatory, by reading the names of these services itself you can figure out what they do.
11. The interesting thing is that these services don't communicate with the database, only the DAL has the right to communicate with the database, so our services communicate with the DAL layer and then they, in turn, communicate with the database and perform CRUD operations on it.
12. Then we have to know how these services work, first, let's talk about the save gossip service, they receive the request body as an argument and the buffer of the image if provided, then they communicate with bad words filter service to sanitize any bad words if there are any and then the save gossip service checks if any image was provided, if it is, then it communicates with the save image service to save the image, in the save image service it communicates with the image moderation service, inside this image moderation service we are using the nsfw.js module, which will detect if there are any adult-rated images and this nsfw.js library uses tensorflow.js ML/AI-related module which is trained with a lot of images or models under the hood to detect any adult-rated images and returns the probability of having those kinds of images and then we ourselves have set up the serenity level or the level of probability that we allow, if adult-rated images are detected then we throw a 400 bad request status code along with the specified error and if no adult-rated images were detected then our save image service communicates with our save image proxy , in here we have adopted a proxy/circuit breaker pattern in our program , which communicates with our save image DAL layer and then the dal layer stores it in the image Kit external asset management service , if for some reason that fails then it will be stored in backup storage that is cloudinary asset management service, then save image service returns the details of the stored image like fileId and the service used as storage etc.., to the save gossip service and then save gossip service sends all the data to the save gossip DAL layer which saves the gossip and sends the saved gossip data back to the save gossip service and then once the data is successfully stored, then we check whether any link was provided , if there was any , then the details of the gossip along with link is passed as an argument to the the publishers, and the 200 success status code is sent back to the client and the gossip is available to be viewed, but our publisher has the connection to our message queue, the message queue is a setup that we have implemented using a service called as RabbitMQ message broker, which helps us to create connection to the RabbitMQ server and allows us to create channel in them and then create queues in them so that we can enqueue messages in them , then those messages are consumed by the consumers and these consumers communicates with our malicious link detection service which uses external link detection service called malicious URL Scanner and if the the link is detected to be unsafe then the consumer communicates with our other publisher called delete gossip which enqueues the message in the delete gossip queue which will be consumed by our other consumer and that consumer will communicate with the delete gossip service than that will communicate with the DAL layer for that specific gossip to be deleted along with the image.
13. We are also doing rigorous unit/integration testing and regression testing using libraries like jest and super test.
14. we have configured a rate limiter which is based on a token-bucket algorithm to protect our web server from attacks like DDoS, brute force, etc.., each and every IP address will be provided with 3 tokens, and 1 token is spent on one request, so the single client can send only 3 requests per minute and after that, they will not be able to send any requests and these tokens will be refilled to 3 tokens after a minute.
15. we have configured to use mature loggers like Winston and to get the details on the request, we are using the express-useragent library, and then we are using centralized log collection tool, called datadog, using that we can collect the logs and visualize and analyze them to extract more info on the behavior of our system in different conditions and when used by different clients.
16. Posting of gossip can be either of the 4 combinations, either it can consist only the gossip or gossip + link or gossip + image or gossip + link + image.
17. On average, posting only gossip takes around 10 milliseconds, and posting gossip+link takes around 11 milliseconds, and posting gossip+image depends on the size of the image, we have set the limit of the image size to be <5MB so anything above that will be rejected with an error response, so an image <1MB takes around 1.3 sec and image 1MB> and <2MB takes around 2 sec and any image above 2MB - 5MB takes around 3 sec, and posting gossip+image+link again depends on the size of the image along with that around 1-2ms added for the link.
18. We can drastically decrease the time taken / latency while posting but that comes with a trade-off, and those are, we can cut off the logging, by doing this we can reduce 1-3ms, but if we do that we will not be able to gain insight on each and every request, so it is a necessary evil for observability and monitoring, or we can cut off the image moderation, which takes about 250-300 Ms if there was an image provided, by doing this we will eventually decrease the quality of the content in our platform so that is a big no, or we can take out bad words filter, which takes around <1ms, this is not a huge improvement so not a huge help, and doing this would just decrease the quality of content and all these things would eventually take too long if we hadn't implemented message broker, so implementing message queue where its possible is helping us to decrease the latency and one of the major latency issue or bottleneck is with our external asset management service, which is imageKit.io and cloudinary.com , since every image is stored there , it takes too long for it to be stored in these services, and the solution for this would be store the images inhouse , in our database , but imageKit.io is providing us with various other useful features so we decided that latency is acceptable compared to the features offered , like the auto resizing and decreased image quality and lazy loading of image etc.., , so taking all these things into account , we have setup the things that would be best to our context.
19. To secure our app from various vulnerabilities, we are using a library called a helmet, It's not a silver bullet, but it can help!
20. We are also using a code analysis tool called codacy, which checks code security, quality, duplication, complexity, manageability, etc.., and gives an overview and grades the project.