Category Archives: Web

d3.js Appealing Visualisations

d3.js Appealing Visualisations

d3js appealing visualisations

 

With the ever-increasing amount of data, both in terms of quantity as well as quality, what we need is a precise and accurate way to represent it for better comprehension and facilitate decision making. That’s where d3.js comes to rescue.

d3 stands for Data-Driven Document, i.e. when your web-page is interacting with data. Data can be as simple a simple array of integers or can be as complex as something else.

 

Why choose d3.js?

 

  • it works seamlessly with existing web technologies
  • can manipulate any part of the document object model
  • it is as flexible as the client side web technology stack (HTML, CSS, SVG)
  • takes advantage of built in functionality that the browser has, simplifying the developer’s job, especially for mouse interaction.

 

What d3.js is not?

 

  • it is not a graphics library
  • it is not a data processing library.
  • it doesn’t have pre-built visualizations

 

D3.js is tools that make the connection between data and graphics easy. It sits right between the two, the perfect place for a library meant for data visualization.

 

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation. ~ d3js.org

 

Show me some code…

Simple bar chart

 

<!DOCTYPE html>
<meta charset=utf-8>
<style>
.chart div {
font: 10px sans-serif;
background-color: blue;
text-align: right;
padding: 3px;
margin: 1px;
color: white;
}
</style>
<div class=chart></div>
<script src=http://d3js.org/d3.v3.min.js></script>
<script>
var data_points = [3, 5, 23, 45, 67, 98, 150, 220];
var plot_scale = d3.scale.linear()
.domain(d3.extent(data_points))
.range([5, 420]);
d3.select(.chart)
.selectAll(div)
.data(data_points)
.enter()
.append(div)
.style(width, function(data_point) { return plot_scale(data_point) + px; })
.text(function(data_point) { return data_point; });
</script>
view rawbar_chart.html hosted with ❤ by GitHub
Bar html

 

Well… that’s the only html and does look very nice and professional. We need more power.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Cost-effective and Variable IP address Google Search Crawler

Cost-effective and Variable IP address Google Search Crawler

cost effective and variable ip address google search crawler

 

The task was to create a highly scalable and cost-effective google search crawler. Challenges were:

1. Scalability — Perform maximum google searches in minimum time.

2. Google temporarily blocks an IP address.

3. Minimal Cost.

I will talk about step by step process of how my solution evolved to create a scalable and variable IP address crawler with infrastructure costs as low as $0.021 for 5000 searches.

 

Problem

 

Initially, it seemed straightforward. I quickly wrote a ruby script to perform a google search for different queries sequentially and used Nokogiri to parse the HTML response. This worked well until the time the number of searches was less than 500 (approx).

Once the number of Google search queries increased, the problem was that searching on google sequentially was not scalable. This problem of sequential search can be optimized by running them parallelly using a queue processing software such as sidekiq. But there was another major problem that I was facing here.

Google was blocking my IP address after approx. 500 queries.

It was impossible to scale the application with the approach I was following.

Solution

 

his was a Rails application and AWS was being used.

To tackle the challenge of getting blocked by Google, I used AWS Elastic IPs. I started running the google searches in parallel sidekiq jobs in a single AWS instance and as soon as Google started blocking the instance IP address. I would

1. Allocate a new elastic IP in my AWS account.

2. Disassociate the current elastic IP from the instance.

3. Associate newly allocated IP address with the instance.

4. Deallocate previous elastic IP address.

While this solved the problem of getting blocked by Google. The problem that persisted after this was that, all the sidekiq jobs would stop and wait for the new elastic IP address to be allocated and associated. And there is still a limit on the number of sidekiq jobs that can be run on an instance based on its infrastructure.

The most cost-effective, scalable solution that I found for the above-mentioned problems is mentioned below. Let’s say there are a certain number of Google searches to be done. I followed the following steps:

1. Divide all the search queries in a group of 500 each. (Because google blocks after approx 500 queries).

2. For each group of search queries, create a rake task to run google search for them.

3. Dockerise the application and push it on docker hub.

4. From an AWS instance, start creating micro instances that will be used for google search of 500 search queries. So, I would spawn a t2.micro instance for each group of search queries and pass the queries to it in a user-data script that runs immediately after an instance is launched.

5. Each AWS instance was spawned using Hashicorp Terraform using a prebuilt Amazon machine image (AMI) which I created using Hashicorp Packer.

6. A user-data script is a shell script that you can create to run tasks immediately after an AWS instance is launched. In my user-data script, I created a docker-compose file. And ran docker build using it.

7. After the docker container of my application was up and running, the next task in the user data script was to run that rake task for google search of all the search queries passed to the user-data script.

8. After the google search for all the search queries was complete, I called an API in main instance to destroy current instance.

This is how I would spawn an AWS instance for each group of search queries. Spawning of each instance happened parallelly in a sidekiq job and then call the main instance to destroy itself using terraform-destroy.

 

Cost

 

Each t2.micro instance ran for about 10 min for a google search of 500 queries. The cost of a t2.micro instance is $0.013. That makes the cost of running 500 google searches $0.0021 per instance.

So, if there are 5000 google searches to be done, then there will be 10 instances spawned and the cost for these google searches in total will be $0.021.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

413/414 Request URL/Entity Too Large Error Nginx

413/414 Request URL/Entity Too Large Error Nginx

what is 413/414 request URL/entity too large issue?

 

Ever tried to upload a huge file or send a very large payload in the request?

 

If yes, you must have received status code 413 from nginx saying that the “request entity is too large for nginx to handle”.

The reason for this error response is because of the “client_max_body_size” parameter in the nginx configuration.

Web servers keep a limit on the maximum size of the request that can be sent to them. This is handled using client_max_body_size parameter. That is because, as mentioned in this stack exchange answer and I quote —

If you configure “client_body_max_size" to a large value, then you are putting your server in the same scenario as by 2013 when Django allowed users to use a very long password forcing Django (rather the server where it is hosted) to perform very expensive hash calculations leading, as you may guess, to a denial-of-service attack against the whole Django’s authentication framework.

In nginx documentation, By default, the value client_body_max_sizeis 1 MiB.

If it is a requirement to change client_body_max_size in nginx configuration, It can be done in the following way:

  • Run the following command to editnginx.conf the file.

 

vi /etc/nginx/nginx.conf

 

  • Add the following line at the top of httpserver or location context to edit the size and then save and close the file.

 

client_max_body_size 10M;

 

  • Restart nginx using the following command.

 

sudo service nginx restart

 

There you go, the maximum permitted request size in nginx configuration has been changed.

HTTP 414 request-URI too large

 

 

I encountered this error while working on Open-Source project in elixir— Avia Commerce.

 

The HTTP 414 URI Too Long response status code indicates that the URI requested by the client is longer than the server is willing to interpret.

 

Talking in terms of “Nginx” web server. This can also be handled in a similar manner as HTTP 413 error. To handle this we have to modify large_client_header_buffers parameter in the server configuration.

As mentioned in the documentation, the default size of 

 

large_client_header_buffers is 8 KB.

 

This way you errors pertaining to max size of request payload or request URI can be handled.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

 

Top 5 Free Online Courses

Top 5 Free Online Courses

top 5 free online courses

To make things easy for you in the coming year, we have compiled Top 5 free online courses which can be leveraged and used to cover up the lost time. The resources are meant to provide insight and prepare you for the trends that are considered to be game changers in 2019.

New Year Resolutions — remember those pesky things everyone keeps talking about at the beginning of the year. Everyone looks so hyped talking about what they want to achieve this year. Well now that we are at the brink of 2018, it’s time to look back and reflect on what you thought you would accomplish.

“I am staring into the abyss full of darkness where nothing is visible.” If this is what comes to your mind when you look back at 2018 then you are not the only one. Now now, no need to get depressed and all, pondering over that you have wasted a full year is not so productive. What matters is to remember that there is still one month to make up for the time lost. Well, of course, you can’t cover one year worth of work in just one month, but you can use this one month and prepare for the upcoming journey.

Top 5 free online courses you can take advantage of, before 2018 ends:

1. AWS Certified Machine Learning

 

Let’s begin with something that everyone is talking about, and that is Machine Learning. It’s a free online course with myriad applications and utilities, Machine Learning is changing the way we live and communicate with technology. With AWS certified Machine Learning course take a deeper dive into the machine learning curriculum. AWS offers 30+ digital ML courses totaling 45+ hours of awesomeness. Developers, data scientists, data platform engineers, and business decision makers can use this training to learn how to apply ML, artificial intelligence (AI), and deep learning (DL) to their businesses unlocking new insights and value.

 

 

Search Engine Optimization (SEO) is one thing that affects the visibility of a website or a webpage. SEO can make or break your business, so it is imperative that you invest time in it. With HubSpot’s SEO Training.it’s a free online course, you’ll learn how search engines rank your content, how to measure the state of your current SEO, and how to figure out an approach to SEO that fits with your business’s needs. This course helps you evaluate the state of your SEO authority and identify specific areas for improvement so you can give your website a better chance of ranking.

 

 

As an entrepreneur, if you are not taking advantage of social media than you are missing out on increasing the reach of your business. Still, there are multiple startups and new ventures that are ignoring social media as a powerful tool. Facebook Blueprint eLearning is a series.it’s a self-paced online course that you can take to learn more about advertising on Facebook and Instagram. With more than 90 eLearning courses, Facebook Blueprint helps you learn the skills to move your business forward.

 

 

Unleash the real power of analytics with Google’s Analytics Academy. It’s a free online course focused on helping you learn about Google’s measurement tools so that you can grow your business through intelligent data collection and analysis. From the basic level to pro level, learn all about what data can do for you.

 

 

Email is a crucial channel in any marketing mix, and never has this been truer than for today’s entrepreneur. Curious what to say? How to say it? How often to hit “send”? Learn it all in this free course by Skillshare.
If you are looking to kick-start a new campaign or looking forward to changing your existing email strategy — get started now.

 

With these free and transformational online courses, you can take the first step toward acing your business growth.

So which one would you enroll to? Any other free course that helped you grow your business?

6 Essential Elements Of An App Marketing Strategy

6 Essential Elements Of An App Marketing Strategy

Contrary to popular belief, the core of the success that applies in marketing rather than the design or development of the app. A very simple (but well made) app that is cheap to make, but well marketed, could provide a better return on investment than a more extensive (and expensive) app that has just added to the Market.

App marketing requires a multi-faceted approach that is coordinated and coherent. You have to create a great app and consistently promote it through as many channels as possible for a shot at success in the app store ecosystem.

App marketing is an activity that starts even before the real marketing for the app could start. Right from the planning stage, one needs to get in the shoes of a marketer.

Here, we are going to cover the various opportunities for marketing that app developers can use and increase their ROI.

 

Pre-Development Stage

 

The pre-development stage is the base from where you start your journey. It is said that well begun is half done.

 

1. IDEA

 

Idea

 

The first step we need is an idea that inspires us. If you yourself don’t have a strong belief of the idea, how would you spread the enthusiasm to others?

Think Novel and Think differently. When one idea has already become successful, there is a minimal chance that a clone too will repeat the success. Nobility, innovativeness, and solutions to an existing problem have the best chances to succeed. Hence you should make sure that your app either solves an existing problem or if the problem has already been solved then provide an intuitive or unique solution.

 

2. RIGHT ECOSYSTEM

 

Ecosystem

 

Each actor who is part of your app development process- developer, designer, and even the sales team need to be chosen with due diligence.

Don’t wait until the last minute to find a developer. Start early and take your time to find the right person or firm. Otherwise, it will cost you, both figuratively and literally.

 

3. DESIGN

 

App Designing

 

Create a wow app! Everything else will fall into place. The war of apps is fought on user experience. Make sure that you are well prepared. Develop a small but solid app and launch it. Gather feedback. Once you know what the audience liked and yearn for, plan for phase 2 accordingly. When you focus on excellent design, your product becomes your marketing.

Bonus Tips Tip 1 – Don’t go with the crowd Tip 2 – Your idea should solve an existing problem Tip 3 – Software development requires an odd mix of attention to detail, creativity, and tenacity. It is mentally challenging, and an experienced developer is worth paying a Premium

 

Post-Development Phase

 

App Marketing is considered incomplete without online Marketing. Online marketing can help you reach to a worldwide audience. With up to 100% of your target audience spending most of the time surfing the internet, an online presence and availability become crucial.

 

4. WEBSITE

 

Webiste Design

 

The main reason to have a website is to provide an easy way for the users to find you, learn about you and research about you. There is ample chance that people will hear your app name and then Google it. Here they need to land on a page that makes them feel comfortable and create confidence to download and try the app. Remember you don’t need a giant website for your business, just a simple site that tells about you, your plans and how the app works.

GET STARTED

a. Get professional help

b. Keep it simple

c. Use it for market research

d. Get feedback

 

5. VIRAL VIDEOS

 

Viral Video

 

A successful viral video can help brands engage with consumers on a much larger scale than a TV ad and is much cheaper to produce. Once the video is shared among hundreds and thousands of people, it becomes “viral.” Viral videos are a great way of getting away from the interruption of pushing messages at people and instead of using entertainment and an emotional connection to encourage consumers to pull your message to them.

CHECKPOINTS

a. The video must be entertaining for the viewers

b. Get apt metatag descriptions to go with the video

c. Encourage people to enjoy and share your content

d. Analyze the metrics

 

6. BLOGGING

 

Blogging

 

Blogging about your app is an efficient and effective way to enhance your profile. The blog should usually describe the overall experience behind creating the app and the enhancements you wish to do. It’s a way to communicate your ideas to your customers and engage them in your business. The followers on your blog are the ready market to reach out. SEO can be potentially increased by sharing the blog on various social media sites.

QUICK FACTS

a. Pinterest is the fastest growing social media network and diverts the maximum traffic

b. Facebook gets 83 percent of the target audience for most apps

c. LinkedIn and Quora also contribute to generating leads

Conclusion

 

App Marketing can be a tough hurdle to overcome. When dealing with app marketing, every detail counts. In order to succeed it is advised that one should focus on what users want from your app and develop an app which would provide a feeling of satisfaction to users. By following the points mentioned in the post you can overcome this hurdle. If you are interested in app marketing and really eager to build a product which could be the game changer, try hiring top curated resources from RemotePanda. To know more contact us HERE.

A Quick Overview of HTTP Methods

A Quick Overview of HTTP Methods

a quick overview of http methods

HTTP is the language of the web. If you’ve ever been involved in developing or communicating with a web server, chances are that you’ve been using HTTP.

HTTP is a server-client protocol. All communication is initiated by the client, in the form of an HTTP request. On receiving this request, the server sends an HTTP response back to the client.

Here is an excellent overview of HTTP that covers the structure of communication over HTTP— https://www.jmarshall.com/easy/http/

A simple HTTP request to fetch posts from a server could look like this –

GET /posts HTTP/1.1

 

The first part, GET is the HTTP method. The second part/posts are the URI. The third partHTTP/1.1 is the HTTP version.

 

The HTTP method indicates the action we wish to perform, and the URI indicates the resource that we want to perform the operation on.

HTTP-Model

The HTTP specification defines these methods – GET, POST, PUT, HEAD, DELETE, OPTIONS and a few more.

 

GET

GET requests are used to retrieve information about the resource specified by the URI. GET is a safe method — that means a GET request should not result in any changes in the server state. It should not cause creation, updating or deletion of any application data. It should be used only for ‘read-only’ actions.

 

POST

POST requests are used to submit data to the server. POST requests may contain a data payload to be submitted to the server. The action performed by the server is determined by the server code. POST requests may be used to create a new resource or to submit data for processing.

 

PUT

PUT requests are used to save an object at the location specified in the request URI. PUT requests should be idempotent. That means that if two or more identical PUT requests are received and executed, the result should be equivalent to executing such a request only once. 

To draw an analogy, a=5 is an idempotent operation since running it once or multiple times results in the value of a being 5. In contrast, a=a+1 is not an idempotent operation, since the value of changes based on how many times we execute the operation.

 

DELETE

DELETE requests are used to delete the object at the location specified in the request URI. DELETE requests are also idempotent.

 

HEAD

HEAD requests are used to retrieve just the headers that would be present in the response of an equivalent GET call. It could be used simply to check whether or not a resource exists or to retrieve the Content-Length Header before deciding whether or not to download a large file. You could also check the Last-Modified header to see if the file was modified since you last retrieved it. The HEAD method is safe. All safe methods are also idempotent since doing nothing once has the same effect as doing nothing multiple times.


Conclusion:

Not all POST methods need to be unsafe. We had a case where we needed to send a large number of email addresses to the server and respond with a yes or no for each of those addresses. It seemed like a GET request at first, but we discovered that some browsers and proxies limit the length of a GET request. It also started feeling more like the URI pointed to a processing service and not to an object, and that it would be fine to POST a list for processing.

To know more about client-server protocol or want to hire a back-end Developer you can contact RemotePanda.