Author Archives: Team RemotePanda

About Team RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from our city Pune. All talents associated with us are close network connections. When we connect them with you, we make sure to manage their quality, their growth, the legalities and also the delivery of work. The idea is to make remote work successful for you.
Cost-effective and Variable IP address Google Search Crawler

Cost-effective and Variable IP address Google Search Crawler

cost effective and variable ip address google search crawler

 

The task was to create a highly scalable and cost-effective google search crawler. Challenges were:

1. Scalability — Perform maximum google searches in minimum time.

2. Google temporarily blocks an IP address.

3. Minimal Cost.

I will talk about step by step process of how my solution evolved to create a scalable and variable IP address crawler with infrastructure costs as low as $0.021 for 5000 searches.

 

Problem

 

Initially, it seemed straightforward. I quickly wrote a ruby script to perform a google search for different queries sequentially and used Nokogiri to parse the HTML response. This worked well until the time the number of searches was less than 500 (approx).

Once the number of Google search queries increased, the problem was that searching on google sequentially was not scalable. This problem of sequential search can be optimized by running them parallelly using a queue processing software such as sidekiq. But there was another major problem that I was facing here.

Google was blocking my IP address after approx. 500 queries.

It was impossible to scale the application with the approach I was following.

Solution

 

his was a Rails application and AWS was being used.

To tackle the challenge of getting blocked by Google, I used AWS Elastic IPs. I started running the google searches in parallel sidekiq jobs in a single AWS instance and as soon as Google started blocking the instance IP address. I would

1. Allocate a new elastic IP in my AWS account.

2. Disassociate the current elastic IP from the instance.

3. Associate newly allocated IP address with the instance.

4. Deallocate previous elastic IP address.

While this solved the problem of getting blocked by Google. The problem that persisted after this was that, all the sidekiq jobs would stop and wait for the new elastic IP address to be allocated and associated. And there is still a limit on the number of sidekiq jobs that can be run on an instance based on its infrastructure.

The most cost-effective, scalable solution that I found for the above-mentioned problems is mentioned below. Let’s say there are a certain number of Google searches to be done. I followed the following steps:

1. Divide all the search queries in a group of 500 each. (Because google blocks after approx 500 queries).

2. For each group of search queries, create a rake task to run google search for them.

3. Dockerise the application and push it on docker hub.

4. From an AWS instance, start creating micro instances that will be used for google search of 500 search queries. So, I would spawn a t2.micro instance for each group of search queries and pass the queries to it in a user-data script that runs immediately after an instance is launched.

5. Each AWS instance was spawned using Hashicorp Terraform using a prebuilt Amazon machine image (AMI) which I created using Hashicorp Packer.

6. A user-data script is a shell script that you can create to run tasks immediately after an AWS instance is launched. In my user-data script, I created a docker-compose file. And ran docker build using it.

7. After the docker container of my application was up and running, the next task in the user data script was to run that rake task for google search of all the search queries passed to the user-data script.

8. After the google search for all the search queries was complete, I called an API in main instance to destroy current instance.

This is how I would spawn an AWS instance for each group of search queries. Spawning of each instance happened parallelly in a sidekiq job and then call the main instance to destroy itself using terraform-destroy.

 

Cost

 

Each t2.micro instance ran for about 10 min for a google search of 500 queries. The cost of a t2.micro instance is $0.013. That makes the cost of running 500 google searches $0.0021 per instance.

So, if there are 5000 google searches to be done, then there will be 10 instances spawned and the cost for these google searches in total will be $0.021.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

ProTips Postgres 1: Looking through the Postgres Window

ProTips Postgres 1: Looking through the Postgres Window

protips postgres 1 looking through the postgres window

 

Problem – 

 

We encountered a situation where we had to select only one record for each values of a column (lets say ‘user_id’) ordered by a different column (lets say ‘price’) in a certain table (lets say ‘purchases’). Doing this active-record way was very inefficient. By the active-record way, we had to:

1.Fetch all the user_ids from purchases table

2.Loop through all the user_ids, for each user_id

A.Select once record ordered by price

 

This approach lead us to query the database number of user times + one query for selecting all the user_ids (N + 1 query problem).

 

Solution – 

 

We struggled to find a better solution in active-record for this problem and window functions of PostgreSQL came to the rescue.

To understand Postgres window functions, we must understand GROUP BY clause. We use GROUP BY clause to apply aggregate functions on a subset of rows of a table.

For example, you have a table called “purchases” with the schema as shown below.

 

 

1
2
3
4
5
6
CREATE TABLE purchases (
  purchase_id serial PRIMARY KEY,
  product_id INT NOT NULL,
  price DECIMAL (11, 2),
  user_id INT NOT NULL
);

 

Lets insert data into this table.

1
2
3
4
5
6
7
8
9
INSERT INTO purchases (product_id, price, user_id)
VALUES
(1, 10, 1),
(2, 20, 1),
(3, 30, 1),
(4, 40, 2),
(1, 10, 2),
(5, 50, 2),
(6, 30, 2)

 

To calculate average of all the purchase prices, we will use AVG aggregate function.

1
SELECT AVG (price) FROM purchases;

This returned the average of prices for both users with id 1 and 2.

1
2
3
avg
--------
 27.1428571428571429

 

Now, if you want to find AVG (A postgres aggregate function) of the price of items that a specific user has purchased. You would use GROUP BY clause on user_id. Thus the following statement will return average for each user_id.

1
SELECT AVG (price) FROM purchases GROUP BY user_id;

Above query return the following

1
2
3
4
user_id          avg
--------      --------
   1          20.0000000000000000
   2          32.5000000000000000

 

As you can see,

The AVG aggregate function has reduced the number of rows returned by the SQL query.

The GROUP BY clause groups the rows of the table into a subset of rows based on a column(s).

Similarly, a window function also operates on a subset of rows of a table but it does not reduce the number of rows. It returns the actual records of the table in the output with an extra column “avg” if you have applied AVG function to the prices.

In the query below the AVG function works as a window function that operates on a set of rows specified by the OVER (PARTITION BY) clause.

For example, the below query will return the product_id, user_id, price and average from “purchases” table for each user group.

 

1
2
3
4
5
6
7
SELECT
product_id,
user_id,
price,
AVG (price) OVER ( PARTITION BY user_id )
FROM
purchases

The output

1
2
3
4
5
6
7
8
9
product_id    user_id   price   avg
------------ ---------- ------- -----
     1           1       10.00   20.0000000000000000
     2           1       20.00   20.0000000000000000
     3           1       30.00   20.0000000000000000
     4           2       40.00   32.5000000000000000
     1           2       10.00   32.5000000000000000
     5           2       50.00   32.5000000000000000
     6           2       30.00   32.5000000000000000

 

You can also apply other operations with PARTITION BY clause. Like if you want the results to be returned in order, you can use ORDER BY clause. You can also use built-in window functions like row_number(), rank(), etc.

These built-in window functions add a number to each row based on their order. The row_number() function assigns a serial number to each of the rows returned. So if you want a limited number of records for each user_id, you can use it.

The query below will return only one record for each user_id, decreasingly ordered by their purchase price.

 

1
2
3
4
5
6
7
8
9
10
11
SELECT * FROM
(
  SELECT
  product_id,
  user_id,
  price,
  row_number() OVER ( PARTITION BY user_id ORDER BY price DESC )
  FROM
  purchases
) tmp
WHERE tmp.row_number <= 1

 

In the above query, I have ordered each partition by decreasing price and then added a limit on the number of rows for each partition by using the row_number() function. Below is the output.

1
2
3
4
  product_id   user_id   price   row_number
------------- --------- ------- ------------
     3           1       30.00       1
     5           2       50.00       1

Postgres provides many built-in window functions. Some of them are explained below.

 

RANK() function

 

The RANK() the function assigns ranking within an ordered partition.  If the values of the two rows are the same, the  RANK() the function assigns the same rank, with the next ranking(s) skipped.

1
2
3
4
5
6
SELECT
product_id,
user_id,
price,
RANK () OVER ( PARTITION BY user_id ORDER BY price )
FROM purchases

The output

1
2
3
4
5
6
7
8
9
10
product_id    user_id    price   rank
------------ ---------- ------- -----
   1            1        10.00    1
   1            1        10.00    1
   2            1        20.00    3
   3            1        30.00    4
   1            2        10.00    1
   6            2        30.00    2
   4            2        40.00    3
   5            2        50.00    4

 

DENSE RANK()

 

The DENSE_RANK() function assigns the ranking within an ordered partition, but the ranks are consecutive. In other words, the same ranks are assigned to multiple rows and no ranks are skipped.

1
2
3
4
5
6
SELECT
product_id,
user_id,
price,
DENSE_RANK () OVER ( PARTITION BY user_id ORDER BY price )
FROM purchases

The output

1
2
3
4
5
6
7
8
9
10
product_id   user_id    price   rank
------------ ---------- -------  -----
     1         1        10.00     1
     1         1        10.00     1
     2         1        20.00     2
     3         1        30.00     3
     1         2        10.00     1
     6         2        30.00     2
     4         2        40.00     3
     5         2        50.00     4

There are many other window functions that are listed below.

  1. FIRST_VALUE()
  2. LAST_VALUE()
  3. NTH_VALUE()
  4. PERCENT_RANK()
  5. LAG()
  6. LEAD()

The DENSE_RANK() function assigns the ranking within an ordered partition, but the ranks are consecutive. In other words, the same ranks are assigned to multiple rows and no ranks are skipped.

1
2
3
4
5
6
SELECT
product_id,
user_id,
price,
DENSE_RANK () OVER ( PARTITION BY user_id ORDER BY price )
FROM purchases

The output

1
2
3
4
5
6
7
8
9
10
product_id   user_id    price   rank
------------ ---------- -------  -----
     1         1        10.00     1
     1         1        10.00     1
     2         1        20.00     2
     3         1        30.00     3
     1         2        10.00     1
     6         2        30.00     2
     4         2        40.00     3
     5         2        50.00     4

There are many other window functions that are listed below.

1.FIRST_VALUE()

2.LAST_VALUE()

3.NTH_VALUE()

4.PERCENT_RANK()

5.LAG()

6.LEAD()

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Interview Series: Thomas Budiman, Co-founder of Insight Design on MVP

Interview Series: Thomas Budiman, Co-founder of Insight Design on MVP

interview series thomas

About

Kevin Sahin

Designer at Hanno

Specialties: User Experiences, User Interface Design, Wireframing & Prototyping (FramerJS), Information Architecture, User flows & Stories, Design system, Front-end Development. Design team management.

Contact: http://www.budiman.com

Linkedin: https://www.linkedin.com/in/thomasbudiman

 

Q.1 Why should startups begin with building a Minimum Viable Product?

 

I see the importance of a minimum viable product not only from the speed and cost-efficiency but also the focus. By building the minimum viable version, we can focus on solving the main problem, finding the right solution or opportunity, testing it to the target audience and getting some feedback as a fuel for the next improvement of your product.

 

Q.2 How do you prioritize features for a Minimum Viable Product (MVP)?

 

To prioritize features for MVP, we need to see which features are the backbone. These features need to support our products so they can be used or tested by the market. In addition, we can also use impact-effort quadrants to help us determine which features we need to bring to the next iteration.

impact-effort quadrants

 

Q.3 What is the best advice you can give to budding Startup CEO on developing a successful MVP?

 

In my opinion, we can’t definitively define what the MVP should look like and how. Different products will have different approaches for an MVP, for instance, some people might think of landing pages as their MVP where they can collect emails and early feedback. Others might say you should focus on the functionalities more than the aesthetics. Well, that can’t be true if you think one of your product values is to be nice-looking and unique. So, we should see the MVP as a form of approach for us to move forward (even with baby steps) in product development or starting a new startup towards understanding who our users are, what their needs are and what opportunities exist–without taking a long time and spending big costs.

 

Q.4 How can you save some extra cash when developing an MVP?

 

I think we know that to save extra cash is the gist of an MVP. Well, this needs to be seen in terms of what products you make, what the team looks like, what strengths the founders are and perhaps some other aspects. I see the most important thing is the strength of the founders. If you want to save the budget, founders need to step in and make their hands dirty. If there is a team or other person working on it, the founders also need to at least understand the process,

so that they don’t take steps that result in wasted time and costs. Last but not least, being strategic and planning carefully in every move.

 

Q.5 Is building MVP still useful in 2019 and coming years?

 

Yes, with a better version, of course. As I said on question #3, we can’t definitively define what the MVP is. People might say that should be a minimum loveable, minimum awesome or something. People will continue to make the MVP way better than we think five years ago, but the essence will remain the same.

Interview Series: Kevin Sahin, Co-Founder at PricingBot on MVP

Interview Series: Kevin Sahin, Co-Founder at PricingBot on MVP

interview series kevin

About

Kevin Sahin

Co-Founder at PricingBot, book Author, Indie maker

Contact: kevin@pricingbot.co

Twitter: https://twitter.com/sahinkevin

Medium: https://medium.com/@sahin.kevin

Linkedin:https://www.linkedin.com/in/kevin-sahin-89700586/

 

Q.1 How do you prioritize features for a Minimum Viable Product (MVP)?

 

There are many ways to do it. One “easy” way is to answer these questions: What is the biggest problem that I’m trying to solve? What is the easiest solution to this problem?

As soon as your core feature is implemented, do a soft launch with a limited set of users, ask for feedback, implement the most requested features, test, learn, iterate!

The most important thing to keep in mind is not to implement features “because it’s cool”.

 

Q.2 How did you build your minimum viable product? What steps did you follow?

 

We built the initial version for PricingBot in two months. It was really basic, it did one thing, and it did it well. We then soft launched it in Alpha with the users that registered on our landing page.

After several weeks of iteration and calls with our initial users, we released the beta on ProductHunt.

It was great because it brought us 2000+ visits on the websites, several hundred sign-ups, and the users had lots of different use cases/insights.

Instead of asking our users if they would be willing to pay for our product, we just froze accounts and added a paywall. Once the first few users started to pay, we knew our product was viable.

I think there is a big difference between someone telling you he’s willing to pay and someone who actually enters his credit card on your website.

 

Q.3 What are the best practices for a Minimum Viable Product?

 

Use the tools you know best. Don’t use a tool because it’s cool or because you want to test technology. Sometimes you don’t even have to create a product to reach your goal.

A simple landing page, a form, a survey is often enough. Once you have validated an interest you can move forward.

 

Q.4 After developing and launching an MVP, how can you define whether your MVP was successful or not?

 

There are different answers to this question because there can be several goals for an MVP. For us, we wanted to 1) Validate that there is a market for E-commerce price monitoring and 2) Get some insights

During our MVP, we learned that users hated configuring their account by themselves, so we created a new onboarding process to help them. We immediately saw our metrics (activation, MAU…) increase. So this was the first success.

So whatever your core metrics are, they should go up during your MVP iterations.

Then the moment we got our first paying customers was another winner!

 

Q.5 Is building MVP still useful in 2019 and coming years?

 

Of course, it is. I feel like there are more and more “early adopters” online, which is your target when you launch an MVP. It’s never been easier to reach those early adopters, thanks to all the online communities, Facebook groups, Slack, online ads… Users also love to play a role in product building, it’s a trend you can see in lots of industries.  

Interview Series: Will Dayble, Founder Fitzroy Academy on MVP

Interview Series: Will Dayble, Founder Fitzroy Academy on MVP

interview series will dayble

 

 

About

Will Dayble

Teacher, recovering tech entrepreneur. Previous I started a few businesses, did some internet stuff.Now founder @ Fitzroy Academy + lecturing @ Monash Uni on entrepreneurship + impact. Not super into formality. In my spare time, I don’t have any. <3

Linkedin:https://www.linkedin.com/in/willdayble/

 

Q.1 Why should startups begin with building a Minimum Viable Product?

 

Because it’s something to do.

It’s easy to get lost in dreaming about beautiful ideas. It’s so fun to talk about how they’re going to save the planet, make you a zillion dollars and be a perfect, brilliant gift to the world.

But while there’s nothing wrong with dreaming, to get something moving you need to do the work. Make something real and show it to people.

Even better, MVPs are fun. They’re small, simple, and easy to throw out. A good MVP is a vehicle for a light, joyful conversation.

 

Q.2 How do you prioritize features for a Minimum Viable Product (MVP)?

 

As your customers what they want!

It sounds really simple, but it’s amazing how many people start building an MVP without talking to the people who will eventually use that product.

Oh and pro tip: Don’t tell people about your idea. I know you want to because it’s exciting, but try to avoid pitching to everyone you meet.

Instead, ask people to tell you a story about the problem they’ve experienced. How did they feel? What did the try? What worked? What didn’t? Who else was involved? How did those people feel?

Those learnings will help you build a better MVP. When in doubt, talk less and ask more questions.

A good MVP is proof that you’ve listened carefully to your customers.

 

Q.3 What is the best advice you can give to budding Startup CEO on developing a successful MVP?

 

Don’t expect your first attempt will be perfect.

It will most likely suck, and that’s half the fun. We think about it like this: Most entrepreneurs build up projects in this order.

    • Shoes
    • Bike
    • Car
    • House
    • Airplane

Your first business (MVP, project, idea, whatever) will earn you enough money for a new pair of shoes. Cool. Throw it out and try again. The one after that gets you enough cash for a new pushbike. Awesome! Wind it up, try again.

Then comes the car, then ten years later – if you’re still hacking away at it – you’ll have the skills, friendships, money, capacity and knowledge to make something that allows you to buy a house! Hooray!

Now very few entrepreneurs actually continue from one stage to the next. They realise it’s just not their style of working, they change careers, life happens. A vanishingly small percentage goes all the way from shoes to jet plane and that’s totally okay.

And by ‘okay’, I mean good okay. Not ‘good enough’ okay, but awesome, happy, best case scenario okay.

It’s infinitely more important to make friends along the way, have fun, and find meaning in the work.

One of the best outcomes of one MVP I personally made, nearly 10 years ago at a hackathon, was a friendship with a guy called Mark. The MVP he and I built together failed miserably. Total failure.

But Mark and I are still friends. We play video games together most nights and he’s helping me move house next month. Huge win, best outcome possible.

Either way, it’s probably dangerous to try jumping from nothing to jet plane.

Start with shoes! 🙂

 

Q.4a) How can you save some extra cash when developing an MVP?

 

Do less stuff. Make each iteration small and immediate.

Have a small team, make each change to the idea tiny. Talk to your customers more often. Don’t spend money unless you absolutely have to. Even if you have a million dollars, pretend you’re working from scratch.

MVPs aren’t about making loads of cash, they’re about learning what works.

Maybe another tip is to remove your ego. Waste normally happens when ego gets in the way. If you’re not pumped up about your own importance, you’re less likely to do what your ego wants and more likely to do what the MVP needs.

Q.4b) The question you didn’t ask: What’s more important than saving cash?

 

Caring about people! Without people, cash is meaningless.

MVPs /lean/fast iterations are a great way to move fast and break stuff, but it’s easy to lose patience for people within that process.

People and products interact, but people are not products, and products are not people!

So for the ‘people stuff’, I enjoy working slowly, patiently, and transparently, with no hidden agendas or end goals. Friendships take years and reputations are built action by action, day by day. There are no destination or version numbers for falling in love or building a community.

When we lose ourselves in the frantic excitement of progress we can forget to nurture the subtle stuff. There’s a time for balance and counterpoint, for reflection and patience.

Go slow to go smooth, go smooth to go fast. 🙂

 

Q.5 Is building MVP still useful in 2019 and coming years?

 

Yes, and sometimes no.

Yes, in the sense that fast iterations with few assumptions get things done quickly. Less time building things and more time showing people what you’ve built will always be a wonderful way to start something.

On the flip-side, however, is knowing that the MVP approach just isn’t suitable for some problems.

You can’t MVP a home renovation. Trust me, I’ve tried. It requires planning, building permits, council and a bunch of other systems that can’t be started and stopped fast enough for the MVP approach to work!

There is a time and place for everything.

In larger systems, you can’t ‘MVP’ your way around a corrupt government, or basic infrastructure, or state agents with bad intentions. Those challenges generally require a slower, broader approach, which is (both sadly and happily) outside the scope of this interview.

MVPs are about being fast.

So go as fast as possible when fast is appropriate, but avoid it when it’s not. Within the big picture, when the large stuff is scary and hard, you can often find little pockets where you can move quickly and kindly, and bring people along the journey of building things together.

I hope that somewhere between ‘shoes’ and ‘jet plane’ of the whole journey I’ll find the wisdom to know when each approach is best.

Hopefully so will you. I’m not there yet!

🙂

Interview Series: George shares on a series of technology startups

Interview Series: George shares on a series of technology startups

interview series george

 

 

About

George Krasadakis

17+ US patents on Artificial Intelligence, Analytics and IoT • 20 years of digital product development – from concept to launch • 80+ innovative, data-driven projects • 10 multinational corporations • 4 technology startups • Founder of ‘Datamine decision support systems’.

Defining and engineering AI-powered products • Leading technology innovation programmes • Extensive experience in Software Engineering, Analytics, and Data Science projects. Views and opinions are my own.

Medium:https://medium.com/@gkrasadakis

Linkedin:https://www.linkedin.com/in/gkrasadakis/

 

Q.1 Why should startups begin with building a Minimum Viable Product?

 

Startups should naturally think in terms of MVPs.

The MVP approach, if applied properly, allows startups to ship their product earlier and satisfy their early customers, by solving their core problem. This way, startups can avoid the development and operational costs of those not-yet-needed features. Startups can make better use of their limited resources, and launch features according to top priorities and needs of their target customers.

Startups need to move fast and follow truly flexible adaptive product development patterns. The experimental nature and the limited resources of the typical early-stage startup require laser-focus and smart prioritization; which is the basis for defining a Minimum Viable Product: the MVP is all about identifying the smallest subset of the product, which can be built first to deliver value to your users, as early as possible.

The Minimum Viable Product is about what to build when – in what order; it is about learning from customers and adapting. It is perfectly aligned with a good startup strategy – it is a great beginning for a startup, a framework to balance excitement and pragmatism.

 

Q.2 How do you prioritize features for a Minimum Viable Product (MVP)?

 

Well, prioritization can become quite tricky and complicated. To assign good priorities you need to combine strategic thinking, product vision, empathy, market insights and also sufficient technology understanding and domain expertise.

My strategy when prioritizing features is based on deeply understanding the problem, the impacted users, the competition and the state of the art. To prioritize wisely, I ensure that – as a product development team – we envision the ‘ideal’ state and that we are able to clearly articulate ‘how to get there’. I emphasize on the need to understand what is already available in the market, along with the opportunities and the constraints of current technologies.

I do believe that having a bold vision and the ability to ‘think big’ is critical for prioritization. If you are aware of the full potential, the ‘big picture’ of your product, prioritization gets easier and more meaningful. In fact, the more features you have to prioritize, the better the final priorities will be: when you have your ‘full product’ described at the feature level, the relative importance of each particular feature becomes more clear; while the risk of ‘opportunity cost’ – from features that you didn’t consider to build – drops. This is why I encourage the product team to define not only the obvious features but also ‘crazy ideas’ and creative ways to solve the problem for our users.

To get the first definition of the MVP, I work closely with the product development team to ensure that each of the features in the backlog get assessed in terms of value to the user, feasibility and cost. This assessment process leverages all the knowledge, the signals and the insights we have — everything we know for the users and their pain-points. The objective is to assign a single value to each product feature, reflecting its importance in solving the problem for our customers; as early and inexpensively as possible.

During this process (which could have many iterations – reviews with stakeholders and refinements), we keep ranking the backlog by ‘importance’. When the prioritization is stable enough, we draw the line separating the MVP from the rest of the backlog. This is the first definition of the MVP.

Depending on the case and the complexity of the product, I might consider additional research techniques to validate assumptions and get more detailed insights from users. Or visualization tools to provide clarity on the product backlog, the priorities and the roadmap.

 

Q.3 What is the best advice you can give to budding Startup CEO on developing a successful MVP?

 

Think as a user; act as an entrepreneur.

‘Thinking as a user’ implies empathy and deep understanding of the market and customer needs. ‘Acting as an entrepreneur’ implies a strategy, wise use of the available resources and responsiveness to signals from the market – along with the right approach to balancing risk. This is the right mindset to have when defining your Minimum Viable Product and your overall product strategy.

As a CEO, you need to ensure that there is a solid, inspiring product vision there; that your product management team and your stakeholders deeply understand the vision and the roadmap. State your assumptions and validate them as soon as you get the right insights and data. Build a data-driven culture but also emphasize critical thinking when interpreting your data. Make sure that your team appreciates the culture of experimentation and values how data-driven decisions can improve your product development efforts.

Use Agile engineering practices, stay connected to your customers, get ready to make pivots when there are strong signals from the market. Step back, look at the product ‘from a distance’; ask for honest feedback from people outside your domain.

Define a solid success measurement framework for your MVP. Make a small investment to build a ‘product performance dashboard’ which automatically quantifies product engagement metrics and user feedback into KPIs vs targets. The measure, interpret, react.  

Or, you can skip all the above and hire a great CPO 🙂

 

Q.4 How can you save some extra cash when developing an MVP?

 

Assuming a good definition of the MVP and also proper execution, you will avoid unnecessary costs by building only what is needed to achieve your short-term goal: to engage with your customers, cover their core needs and learn. With the MVP you are building a smaller first instance of your product; successful execution means smaller engineering/ development and operational costs.

In some cases, the MVP could even generate early revenue streams – which, depending on the business model and the stage of the startup, could prove to be important for further product development and user base growth.

Q.5 Is building MVP still useful in 2019 and coming years?

 

Well, I believe that MVP reflects a particular approach or philosophy for building products – which should and will be there as ‘common sense’ for product development. Moreover, when building digital products in this rapidly changing technology landscape, you must constantly apply critical thinking about the definition and prioritization of product features – or you can easily get distracted and take risky paths.

I strongly believe that – regardless of the actual terminology used – this approach makes perfect sense and will be there as the basis for modern product development.

413/414 Request URL/Entity Too Large Error Nginx

413/414 Request URL/Entity Too Large Error Nginx

what is 413/414 request URL/entity too large issue?

 

Ever tried to upload a huge file or send a very large payload in the request?

 

If yes, you must have received status code 413 from nginx saying that the “request entity is too large for nginx to handle”.

The reason for this error response is because of the “client_max_body_size” parameter in the nginx configuration.

Web servers keep a limit on the maximum size of the request that can be sent to them. This is handled using client_max_body_size parameter. That is because, as mentioned in this stack exchange answer and I quote —

If you configure “client_body_max_size" to a large value, then you are putting your server in the same scenario as by 2013 when Django allowed users to use a very long password forcing Django (rather the server where it is hosted) to perform very expensive hash calculations leading, as you may guess, to a denial-of-service attack against the whole Django’s authentication framework.

In nginx documentation, By default, the value client_body_max_sizeis 1 MiB.

If it is a requirement to change client_body_max_size in nginx configuration, It can be done in the following way:

  • Run the following command to editnginx.conf the file.

 

vi /etc/nginx/nginx.conf

 

  • Add the following line at the top of httpserver or location context to edit the size and then save and close the file.

 

client_max_body_size 10M;

 

  • Restart nginx using the following command.

 

sudo service nginx restart

 

There you go, the maximum permitted request size in nginx configuration has been changed.

HTTP 414 request-URI too large

 

 

I encountered this error while working on Open-Source project in elixir— Avia Commerce.

 

The HTTP 414 URI Too Long response status code indicates that the URI requested by the client is longer than the server is willing to interpret.

 

Talking in terms of “Nginx” web server. This can also be handled in a similar manner as HTTP 413 error. To handle this we have to modify large_client_header_buffers parameter in the server configuration.

As mentioned in the documentation, the default size of 

 

large_client_header_buffers is 8 KB.

 

This way you errors pertaining to max size of request payload or request URI can be handled.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

 

ACTIVE-RECORD or RAW SQL?

ACTIVE-RECORD or RAW SQL?

 

Active-record is an abstraction layer that facilitates the creation, deletion, and use of ORM objects whose data requires persistent storage to a database, This keeps us away from having to think too much about SQL level queries and makes it very easy to work with our data. It gives us a super easy interface that can to do “almost” anything that we can do with bare SQL statements. Apart from the basic crud operations, active-record lets us do more complicated database stuff like pick a group of records based on criteria, order them, join tables, perform mathematical operations, etc.

Active-record pattern is liked by most because of the above-mentioned reasons. But using active-record solely may not help when your application scales. For example, after using active-record in my application for over a year, the fact that active-record does not have a feature to bulk insert into the database natively has bugged me. Of course, we can use gems to do that, but I personally do not like the idea of using gem for a specific purpose of the bulk insert.

Instead, I like to use raw SQL statements in such cases. SQL statements to bulk insert are much faster than active-record way of doing the same task. I am going to talk about an example when using raw SQL statements optimized tasks that took hours and minutes to complete got finished in a few seconds.

I am using active-record with rails in my application.

In my application, a user can import a CSV file into a database table directly from UI. This CSV file had only one column “email”. Earlier this CSV file had not more than 5k – 6k columns. But last week a user uploaded a file of 842k emails and the systems froze. The request timed out on UI. It was taking a lot of time (HOURS) for this file upload. Imagine this situation in a user’s perspective. The user would be super confused and for us, it was a nightmare.

The problem here was –

 

Our code to upload CSV in the system was not very optimized. We were loading all the data from CSV into the memory and then iterating over it and creating database records one by one using active-record. This made the system to fall apart.

ACTIVE RECORD or RAW SQL – Vikram Singh Jadon – Medium

 

To improve, we rectified one of the biggest mistakes that we were doing i.e. loading all the CSV data into the memory. So we added the code to load one CSV record in the memory and saving it in the database one by one. But this was still taking hours. Because it was sending a query to create a record in the database for each csv row. That means 842k queries to the database.

ACTIVE RECORD or RAW SQL – Vikram Singh Jadon – Medium (1)

 

Active-record does not provide support for bulk import. This is one of the reasons why active-record pattern doesn’t scale very well. So, this time we started writing raw SQL queries for bulk import. We could have done the bulk import for all the 842k records which would have done all the work in not more than 2–3 sec, but raw SQL query would require us to build this query with 842k records in the memory. So, being sceptical about memory consumption, we decided to do it in batches of 5k. So, for every 5k emails, we built the query to bulk insert in the database and as we expected, the time it took to import 842k records in the database was 23sec. (That too because we did in batches of 5k). This time is still acceptable to the user on UI.

ACTIVE RECORD or RAW SQL – Vikram Singh Jadon – Medium

 

The other problem that we faced was that we were using elasticsearch’s scroll API to search results from our database and add those into a separate table. But doing this inside every elasticsearch’s scroll caused the scroll to time out even when the scroll timeout was 5 minutes, which means execution time to add data (this data was of about 10k records) into a separate table was more than 5 minutes. So we needed to add data in a separate table using the database’s bulk import raw query and then running after the update, after saving tasks in the background. This solved our problem and reduced the time for adding data from more than 5 min to 2 sec.

We can use bulk imports, bulk update raw SQL queries carefully wherever we do not require to do the callbacks jobs instantly. So, we can move them in the background. Database SQL queries are fast and efficient. In my personal opinion, we should leverage them wherever we can.

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

How To Integrate WYSIWYG Editor in Elixir/Phoenix application

How To Integrate WYSIWYG Editor in Elixir/Phoenix application

 

What is a WYSIWYG Editor?

 

Gmail’s composing email editor shown below has various tools such as Bold, Italic, Bullets, Paragraphs, underline, Image attaches, link, etc. for users to format text. This enables the sender to format text in the email exactly the same as he wants the receiver to see it without getting into the markup.

Gmail

 

Formatting text in traditional text fields is not possible and hence the text is rendered in a single line which does not look good.

As the name suggests, What You See Is What You Get (WYSIWYG) editor returns the exactly formatted text as the user had added. These editors conceal the markup and allow the Web page developer to think entirely in terms of how the content should appear.


In this tutorial, we will learn how to integrate Classic CK Editor 5 in an elixir form.

CKEditor is an HTML WYSIWYG editor and is open source.

 

How to integrate it?

 

CKEditor’s build is available in a CDN (Content Delivery Network). It is hosted on servers spread across the globe. Therefore, its scripts are loaded faster because they are served from the nearest locations to the end user.

If the same version of CK Editor is downloaded on a browser (even from a different website), it is loaded from cache. So, it speeds up the next requests.

Downloading the editor can be done by using <script > tag.

<script src=https://cdn.ckeditor.com/4.11.1/standard/ckeditor.js></script>

 

Create Classic Editor

 

Suppose, we need to add CKEditor to the description field of a form to edit product details. The id of textarea tag is product_description .

The classic editor is added to the form after the page loads and below script runs.

 

div class=form-group row >
<%= textarea_input f, :description, nil, is_horizontal: true, description: desc., rows: 10 %>
</div>
<script>
ClassicEditor
.create( document.querySelector( #product_description ) )
.catch( error => {
console.error( error );
} );
</script>

 

To remove any tools from the editor, we can add tool names to removePluginsin the .create method. Below code removes ‘ImageUpload’ and ‘MediaEmbed’ tools from the CKEditor.

 

<script>
ClassicEditor
.create( document.querySelector( #product_description ), {
removePlugins: [ ImageUpload, MediaEmbed ]
} )
.catch( error => {
console.error( error );
} );
</script>
view rawremove_tools.eex hosted with ❤ by GitHub

 

We also have to stop HTML textarea editor to show up before CKEditor is loaded in its place. To do that, we will have to make the textarea hidden as shown below by adding class: ‘invisible’ to the textarea tag.

 

<div class=form-group row >
<%= textarea_input f, :description, nil, is_horizontal: true, class: invisible, description: desc., rows: 10 %>
</div>
view rawhide_textarea.eex hosted with ❤ by GitHub

 

Once we edit in the CKEditor and submit the form. The editor sends the edited text along with HTML markup to the backend in string format. The edited text in the backend is saved with HTML markup.

For example: In Aviacommerce, we saved the description of jewelry in bold as shown below.

 

Description of Product in Aviacommerce

 

The description for this product in backend got saved with HTML markup as shown below.

WYSIWYG Editor Integration in Elixir Phoenix – Aviabird – Medium

 

With this setup, we are ready to serve the front-end with HTML markup we have stored in the backend record thisdescription case.

 

Front-end Rendering

 

Let’s discuss how the rendering of HTML happens here. Description containing HTML markup is retrieved from the backend. This is fed to the editor with innerHTML element which renders all the HTML in the description.

With these kinds of editors integrated, applications are subjected to XSS attacks. For example, Aviastore which is a demo application of Aviacommerceexperienced this kind of XSS attack when a user tried to input <script> into an editor. Therefore, making the innerHTML run contents of the script tag. Luckily we had anticipated this and handled it on our side.

XSS attack attempt on Aviastore

 

In AngularSpree also, we created this kind of editor where we wanted to allow innerHTML to be executed along with preventing the XSS attacks. Therefore, we handled this by adding sanitizeHTML pipe . (Click to go to the code) The sanitizeHTML pipe of Angular lets you desanitize HTML only while keeping other sanitization as it is, thereby preventing XSS attacks.

But, to our relief, we do not have to worry about these things with CKEditor. CKEditor handles it on its own. For example, I tried to add some text in bold along with a script in <script> tag. Below is the output.

 

CKEditor handling sanitization

 

Above Image shows that CKEditor is handling the rendering of HTML markup while preventing XSS attack by sanitizing malicious script.

You can refer to this cool blog where DomSanitizer in Angular is explained.


 

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

 

Global Work From Home Day

Global Work From Home Day

It’s time to celebrate the most preferred lifestyle of all professionals, Remote Work! It’s not just a trend anymore, it’s a way of life. People all around the world are being pulled to this amazing concept and it’s working wonders. So to appreciate remote work we are organizing Global Work From Home Day on April 10th in association with Remote-How.

We have been advocating RemoteWork for more than a year now, our platform www.RemotePanda.com has been trying to bridge the gap between developers in Pune and the business in the United States. During this wonderful journey, we got to meet a lot of entrepreneurs who had nothing but praise for remote work and who have been implementing it for a long time. We’ve had our chance of meeting like-minded people and it’s your chance to meet and network with all the fellow remote workers under the hashtag #WorkFromHomeDay.

And to boast off about how cool your remote work setup is and how much you enjoy working remotely, you can post a picture with #WorkFromHomeDay on April 10th.