Author Archives: Team RemotePanda

About Team RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from our city Pune. All talents associated with us are close network connections. When we connect them with you, we make sure to manage their quality, their growth, the legalities and also the delivery of work. The idea is to make remote work successful for you.
why you should not hire a remote employee in 2019

why you should not hire a remote employee in 2019

remote-employee-in-2019

 

Everybody in an organization wants for their business to grow as fast as possible. Many must be thinking about adding a few more skilled employees to their arsenal, and the idea that must be popping up in your mind is “Should we go for hiring Remote Workers?”. Let’s explore the pros and cons of this decision.

“Isn’t hiring someone local more convenient?”. There is an advantage to meeting a person face to face, wandering around their desks and checking in on what they are doing and how much of the work they have actually completed. Catching up with them on coffee breaks and water cooler talks, engaging in interesting (sometimes futile) conversation. You can’t do any of that fun stuff with a remote team, can you?

Well, you must have read a lot of blogs telling you all about why you should hire remote workers, let me tell you why you shouldn’t.

 

1. I would rather hire from a small local pool of talent

The Internet has led to the rise in remote workers becoming integral parts of organizations. The ease of communication and various collaboration tools have further led to ease in acquiring talented employees from all over the world.

But why to look for someone who is physically distant when you can hire someone who you can meet with every day?

Granted, this local person might not be as skilled as the remote workers, but at least you have the benefit of being around him, checking in on him whenever you want, addressing and solving the issues he might be facing. To do all this with your remote team, you’ll have to have access to a simple collaboration tool, a communication channel.

 

2. I hate having extra money to reinvest into my business

Yes, it’s true, hiring remote workers can save you a lot of money. You don’t have to worry about the employee’s desk, computer or any other thing that is needed to work efficiently: remote workers take care of themselves.

But if you hire a local employee, you have the chance to go furniture shopping, buying a brand new, high tech computer sounds so fun, and you wouldn’t want to miss out on that, would you? And if you build a team big enough that it can’t possibly fit in your current workplace, you have the option of finding a new and bigger place!

Remember the scene from Batman where Joker burns down a whole pile of cash? You can enjoy a very similar feeling.

 

3. I don’t like increased productivity 

According to a recent survey, remote workers are more productive than the in-house team.

Related: Remote Teams Vs In-house Teams

You know what happens when employees become more productive right? Goodbye timepass!

Feeling bad that you aren’t getting more time to have those fun, futile conversations with your team? Tired of seeing your employees pour their heart into their work and delivering the best results and that too at the expense of the precious fun time? It’s a blessing that the in-house team isn’t as productive all the time as the remote team. I mean they are productive no doubt, but they can always take some time out during important work to have some fun.

4. I prefer a higher turnover rate

If you treat your remote employees right, they are the most loyal employees you will ever come around, which reduces your employee turnover rate. But do you really want to be stuck with the same old faces for the rest of your life? Hiring new employees is a good thing, meeting new employees, introducing them to your company culture, training them, and teaching them new skills is such a great activity to bond, and definitely, one to look forward to.

5. I don’t care about global warming

Who doesn’t enjoy a long commute? Bring on the one hour journey to my office in heavy traffic, turning up the music as loud as I can (to drown the honks), burning up gas, and causing pollution. More of that, please!

Remote employees miss out on the fun journey; they work from home or a coworking space. Or a beach. They don’t get to turn up the music in their cars or contribute to the already clogged roads.

6. I don’t mind employees taking unexpected vacations and sick days

Remote employees tend to take fewer leaves of absences than the in-house team. Sounds boring right? Being available all the time to work, delivering projects before their deadline, being productive and proactive.

The real fun is in taking a vacation, turning off all modes of communication with your office, and just lying on the beach sipping on a pina colada. We can always get back to work after a good time spent away from it.

I wonder how digital nomads manage to work and travel all at the same time; I find it dreadful to even look at my laptop while I’m on vacation. Who cares about work, right? We’ll be at the office after a couple of days doing the same thing.

Well, I’ve run out of sarcasm, and I hope I have given you enough reasons to start 2019 with remote hiringWhat I described above sounds like a complete disaster.

That’s because remote work is the future of work. You already know all the benefits of having a remote team, such as increased productivity, reduced turnover, reduced leaves of absence, etc. But if you’re still unsure, then you can start with one engagement first, and then keep building more efficiency than an in-house team.

If you need to know the current trends of remote work and how to manage a remote team, we have an entire report dedicated to it; you can DOWNLOAD IT HERE.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Politics in Remote Teams

Politics in Remote Teams

 

One of the most despised thing in a co-located office is the politics which doesn’t seem to cease. People turn to remote work for various reasons, one of them being fewer politics and more focus on the work, but does this really hold true? Is remote team completely free of politics and bias as compared to the in-house team? Remote workers are humans, after all, they are driven by their own passion, goal, ego and more of a mix of all the negatives and positives.

In this whole charade of politics, what the employees don’t understand is that they are disrupting the amazing culture which has been inculcated over the years through tremendous hard work and fun activities. Such a blow to the company culture not only affects the management but also affects the profitability of the company. Now we usually think that politics only happen when the employees are co-located, being physically distant might be the trick to avoid it, but let’s see how remote teams feed the fire that can give rise to this devastating phenomenon.

 

1. Cross-Cultural Differences:

 

When I was interviewing CXOs and developers for our#MakeRemoteWork survey report, I met a developer in the US. She had tremendous experience of working remotely and had been through this roller coaster before.

 

Cultural-Differences

 

She told me that one of the reasons why it happens is that a team has members from different culture and background, so people sharing similar values get along really well, while those who don’t, feel a bit ostracized. There is still a bias towards, race, religion, country and what not.

These are the things that can break an extremely strong team, what employees need to realize is that they are here to work in synergy and harmony not to judge a person on personal traits and discriminate.

 

2. Less socialization:

 

Slack is the virtual office for all the remote workers. You see less of their faces and more of their text messages. Remote workers are generally so involved in their work that they barely talk about their personal lives on the communication channels and talk more about work.

Less-socialization

 

In this process, nobody gets to know what the person on the other side is like. We tend to form a perception about that person based on his work and not for the person he is. This is usually one of the reasons that remote employees feel less engaged in activities other than work, which leads to a difference of opinion between the team. So what’s really important here is COMMUNICATION. On a positive note, remote team members are not great on an emotional level compared to their in-house counterparts, so their efforts to influence internal politics are pretty low.

 

3. Ego Clashes:

 

Shoutouts and all are cool, but it sucks when a kissass employee gets it over a kickass employee.

 

Ego-Clashes

 

It’s true that rewarding an employee boosts the morale of not just employee but sets a standard for other employees as well, but when you’re working with a remote team, you should take into consideration the entire team and not just a single member. In the end, everybody is codependent and desires to be appreciated equally. A team is like a Bad Boys movie, we ride together, we die together, bad boys for life


Now that we know what gives rise to politics in remote teams let’s go through some tips to diffuse these politics.

 

1. Company culture:

 

It’s the responsibility of the management to enlighten their employees about the culture that has been established in the company.

 

 

Company-Culture

 

You shouldn’t let employees learn more about the company through water cooler talks and gossips, this won’t do anyone any good, it will only lead to further misunderstanding of how the company works. So it’s the management or the leader’s responsibility to show his employees the ropes.

 

2. Communication:

 

Organizations who embrace remote work have advanced technology to make communication easier and seamless within the team,

 

 

Communication

 

it’s imperative for the team members to use these communication tools, not just for work, but to stay in constant touch and build a better team as well as company culture.

 

3. Focus on work:

 

It’s no surprise that we are at an organization to work in unison to achieve personal as well as the organization’s goal, so, one of the best things one can do to stay away from all this politics is to focus on the task at hand.

 

Focus on work

 

At the end of the day, everyone wants to feel satisfied and appreciated for the work they have done, and it will only happen when the focus is less on politics and more the work.

 

4. Build Trust:

Trust is one of the most important factors behind every team and organization’s success.

Build-Trust

 

The organization should plan certain team building exercises which would revolve around making long term or short term strategies and also creating a stronger bond during the entire process. This is a win-win situation for the team as well as the organization.

 

Conclusion

 

Politics within a remote team is quite different than the one you’ll encounter in the office, there’s no backstabbing or the usual office drama, although certain cases persist like taking credit for somebody else’s work. It eventually comes down to how well a manager can encourage healthy communication between his team and how well a team can harbor a feeling of trust and respect. 

How to Create a Github Repository from the Command Line

How to Create a Github Repository from the Command Line

how to create a github repository from the command line

 

Git is a great version control system and Github is superb hosting service for git based repositories.

Github provides a nice web interface to create (blank) repositories at the start of the project. But why visit github.com to create a blank repository, so here’s a simple bash script to make this simple task even simpler.

 

git-create(){
repo_name=$1
dir_name=`basename $(pwd)`
if [ $repo_name = ]; then
echo -n Repo name [$dir_name]?:
read repo_name
fi
if [ $repo_name = ]; then
repo_name=$dir_name
fi
username=`git config user.name`
if [ $username = ]; then
echo -n Could not find username, run ‘git config –global user.name <username>’
return 1
fi
token=`git config user.token`
if [ $token = ]; then
echo -n Could not find token, run ‘git config –global user.token <token>’
return 1
fi
echo -n Creating Github repository ‘$repo_name‘…
curl -u $username:$token https://api.github.com/user/repos -d {“name”:”$repo_name“} > /dev/null 2>&1
echo Done.
echo -n Adding remote…
git remote add origin git@github.com:$username/$repo_name.git
echo Done.
}
view rawgit-create.bash hosted with ❤ by GitHub

 

Script is based on Curl and GithubApi.

 

Add this to bash_profile and reload it. Done 🙂
Use git-create to summon 146822610729350.

Be sure to configure GitHub username and access_token in global git configure file.

Hint:

git config --global user.name <username>

git config --global user.token <access_token>

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

CSRF and RAILS protect from forgery

CSRF and RAILS protect from forgery

csrf and rails protect from forgery

 

Cross-site request forgery, also known as a one-click attack or session riding and abbreviated as CSRF or XSRF, is a type of malicious exploit of a website whereby unauthorized commands are transmitted from a user that the website trusts. Unlike Cross Site Scripting (XSS), which exploits the trust a user has for a particular site, CSRF exploits the trust that a site has in a user’s browser. Let’s take a look at the schematic of the CSRF

 

CSRF Scheme

 

 

  • Step1: The Victim connects to secure Bank websites and logs into his account.
  • Step2: A cookie set in the Victims browser containing the session id of the victim.
  • Step3: Victim trips into visiting a malicious page.
  • Step4: Victim receives an html page containing the malicious hidden form.
  • Step5: A web request is executed from the victim’s browser carrying the context of the cookie set in Step2.
  • Step6: Bank Server completes the web requests.

 

Now we know what CSRF is, let’s see how Rails help prevent CSRF.
As Rails uses MVC architecture, Controller actions are protected from Cross-Site Request Forgery (CSRF) attacks by including a token in the rendered html for your application. This token is stored as a random string in the session, to which an attacker does not have access. When a request reaches your application, Rails verifies the received token with the token in the session. Only HTML and JavaScript requests are checked so this will not protect your XML API (presumably you’ll have a different authentication scheme there anyway). Also, GET requests are not protected as these should be idempotent. The requests are validated using the following piece of code

 

def verified_request?
!protect_against_forgery? || request.get? || request.head? ||
form_authenticity_token == params[request_forgery_protection_token] ||
form_authenticity_token == request.headers[X-CSRF-Token]
end
view rawverified_Request.rb hosted with ❤ by GitHub

 

This can be enabled with the protect_from_forgery method, which will perform the check and handle unverified requests, if the token doesn’t match. And it will add a _authenticity_token parameter to all forms that are automatically generated by Rails. It is recommended that this method is added in your ApplicationController, and later on, you can skip it in other controllers if not required.

With all this in mind lets take a look at Rails source code.

 

class ApplicationController < ActionController::Base
protect_from_forgery
end
def protect_from_forgery(options = {})
self.request_forgery_protection_token ||= :authenticity_token
prepend_before_action :verify_authenticity_token, options
end
def verify_authenticity_token
unless verified_request?
logger.warn Can’t verify CSRF token authenticity if logger
handle_unverified_request
end
end
def handle_unverified_request
reset_session
end

 

From the code, we figure out, CSRF protection resets session and lets the request through when CSRF token verification fails.
This in itself is a CSRF vulnerability since it allows anyone to logout users by directing their browser to a page that requires CSRF protection

With Rails 4 application, the nowApplicationController passes a parameter to.protect_from_forgery

 

class ApplicationController < ActionController::Base
# Prevent CSRF attacks by raising an exception.
# For APIs, you may want to use :null_session instead.
protect_from_forgery with: :exception
end
def protect_from_forgery(options = {})
self.forgery_protection_strategy = protection_method_class(options[:with] || :null_session)
self.request_forgery_protection_token ||= :authenticity_token
prepend_before_action :verify_authenticity_token, options
end
def verify_authenticity_token
unless verified_request?
logger.warn Can’t verify CSRF token authenticity if logger
handle_unverified_request
end
end
def handle_unverified_request
forgery_protection_strategy.new(self).handle_unverified_request
end

 

This raises an exception when an unverified request is encountered. The same behavior can be achieved with Rails 3 by overriding the default handle_unverified_request method.

 

Conclusion-

 

Banking server failed to verify the validity of the web request and hence executed it without the victim’s knowledge.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Clean Validations with Custom Contexts

Clean Validations with Custom Contexts

clean validations with custom contexts

 

Active Record validations are well-known and widely used in Rails.

 

class User < ApplicationRecord
  
validates :name, presence: { message: "must be given please" }

end

 

This runs the validation on save, both when creating a new record or when updating an existing record.

on option allows control over when to run the validation, commonly used with value of create or update

 

class User < ApplicationRecord
  belongs_to :club, optional: true 
  validates :name, presence: { message: "must be given please" }, on: :create
  validates :club, presence: { message: "must be given please" }, on: :update  
end

 

This allows creating users without associating them with a Club but enforces the presence of Club on subsequent updates. This pattern is commonly used to allow users to signup with bare minimum form fields and then forcing them to update their profiles with more information on subsequent visits.

Value for the on option is not limited to create and update, we can have our own custom contexts. Like in a multistep form, we can have validations for each of the steps. on options makes this really easy to do

 

class User < ApplicationRecord
  validate :basic_info, on: :basic_info
  validate :education_details, on: :education_details
  validate :professional_info, on: :professional_info

  private
  def basic_info
    # Validation for basic info, first_name, last_name, email
  end

  def education_details
    # Validation for education_details
  end

  def professional_info
    # Validation for professional_info
  end
end

 

In the controller

 

class UsersController < ApplicationController
  ...

  def update_basic_info
    @user.assign_attributes(basic_info_params)
    @user.save(:basic_info)
  end

  def update_education_details
    @user.assign_attributes(education_details_params)
    @user.save(:education_details)
  end

  def update_professional_info
    @user.assign_attributes(professional_info_params)
    @user.save(:professional_info)
  end

  private
  def basic_info_params
    # strong params
  end

  def education_details_params
    # strong params
  end

  def professional_info_params
    # strong params
  end
end

 

With Rails 5 adding support for multiple contexts, we can use multiple contexts together

 

@user.save(:basic_info, :professional_info)

 

This seems pretty neat, let’s go a step further and do this with update_attributes. In current implementation of Rails,
update_attributes does not support validation contexts. We can get around this by defining our own custom method

 

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  def update_attibutes_with_context(attributes, *contexts)
    with_transaction_returning_status do
      assign_attributes(attributes)
      save(context: contexts)
    end
  end
end

 

In the controller

 

@user.update_attibutes_with_context({first_name: 'fname'}, :basic_info)

 

Lastly, we can use with_options to group multiple validations within a context

 

with_options on: :member do |member_user|
    member_user.validates :club_name, presence: true
    member_user.validates :membership_id, presence: true
  end

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Android Permissions

Android Permissions

android permissions

 

To protect the system’s integrity and the user’s privacy, Android runs each app in a limited access sandbox. If the app wants to use resources or information outside of its sandbox, the app has to explicitly request permission. Depending on the type of permission the app requests, the system may grant the permission automatically, or the system may ask the user to grant the permission.

 

Declaring Permissions

 

Declare that your app needs permission by listing the permission in the App Manifest.

Depending on how sensitive the permission is, the system might grant the permission automatically, or the device user might have to grant the request.

For example, if your app requests permission to turn on the device’s flashlight, the system grants that permission automatically. But if your app needs to read the user’s contacts, the system asks the user to approve that permission.

Depending on the platform version, the user grants the permission either when they install the app (on Android 5.1 and lower) or while running the app (on Android 6.0 and higher).

 

<manifest xmlns:android=http://schemas.android.com/apk/res/android
package=com.example.myapp>
<uses-permission android:name=android.permission.WRITE_EXTERNAL_STORAGE />
<uses-permission android:name=android.permission.ACCESS_FINE_LOCATION/>
<uses-permission android:name=android.permission.ACCESS_COARSE_LOCATION/>
<application …>
</application>
</manifest>
view rawAndroidManifest.xml hosted with ❤ by GitHub

 

Requesting Permissions

 

Beginning in Android 6.0 (API level 23), users grant permissions to apps while the app is running, not when they install the app.

 

 

requesting app permission

 

It gives the user more control over the app’s functionality; for example, a user could choose to give a camera app access to the camera but not to the device location. The user can revoke the permissions at any time, by going to the app’s Settings screen.

System permissions are divided into two categories

 

  • Normal permissions do not directly risk the user’s privacy. If an app lists normal permission in its manifest, the system grants the permission automatically. The complete list of normal permissions can be found here.
  • Dangerous permissions can give the app access to the user’s confidential data. For these permissions, the user has to explicitly give approval to the app. The complete list of dangerous permissions can be found here.

 

Handling Permissions at Runtime

 

For applications to support new Runtime Permissions, in `grade.build` file  set `compileSdkVersion`  and `targetSdkVersion` to 23.

 

apply plugin: com.android.application
android {
compileSdkVersion 23
buildToolsVersion 23.0.3
defaultConfig {
applicationId com.example.myapp
minSdkVersion 10
targetSdkVersion 23
versionCode 1
versionName 1.0
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile(proguard-android.txt), proguard-rules.pro
}
}
}
dependencies {
compile fileTree(dir: libs, include: [*.jar])
compile com.android.support:support-v4:23.3.0
compile com.github.bumptech.glide:glide:3.7.0
compile com.google.android.gms:play-services-location:9.2.0
testCompile junit:junit:4.12
compile com.android.support:appcompat-v7:23.3.0
}
view rawbuild.gradle hosted with ❤ by GitHub

 

Moving forward, in <ActivityClass>.java, before accessing any restricted content we will need to check for appropriate permission at runtime. This is to be done using Android `checkSelfPermission` API.
`int hasWriteContactsPermission = checkSelfPermission(Manifest.permission.WRITE_CONTACTS)`

If permission is granted, we are good to go else we need to request permission.

We will need to explicitly call `requestPermissions` API to show the request permission dialog box, Android will not do this for us.

 

private final int WRITE_EXTERNAL_STORAGE_REQUEST_CODE = 123
private void takePhoto() {
//Make sure we have permission to write to external storage
int hasWriteExternalStoragePermission = checkSelfPermission(Manifest.permission.WRITE_EXTERNAL_STORAGE);
if (hasWriteExternalStoragePermission == PackageManager.PERMISSION_GRANTED)
captureImage()
else
requestPermissionWriteToLocalStorage();
}
private void captureImage(){
//Code to capture image from camera
}
private void requestPermissionWriteToLocalStorage(){
requestPermissions(new String[] {Manifest.permission.WRITE_EXTERNAL_STORAGE}, WRITE_EXTERNAL_STORAGE_REQUEST_CODE);
}
view rawcaptureImage.java hosted with ❤ by GitHub

 

In line 5, we get current status for permission. If permission is granted, we continue with our action else we request the user for permission.

 

Handling Permission Results

 

Permission results are communicated to the app via onRequestPermissionsResult callback method. Override this to handle the result.

 

@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
if (requestCode == WRITE_EXTERNAL_STORAGE_REQUEST_CODE){
if (grantResults[0] == PackageManager.PERMISSION_GRANTED)
captureImage();
else{
if (shouldShowRequestPermissionRationale(Manifest.permission.WRITE_EXTERNAL_STORAGE))
showRationaleDialog();
else
Toast.makeText(MainActivity.this, You need to allow permission to Write to External Storage, Toast.LENGTH_LONG).show();
}
}
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
}
private void showRationaleDialog(){
new AlertDialog.Builder(MainActivity.this)
.setMessage(Application needs access to READ/WRITE LocalStorage to store images.)
.setPositiveButton(OK, new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int which) {
requestPermissionWriteToLocalStorage();
}
})
.setNegativeButton(Cancel, null)
.create()
.show();
}

 

If permission is granted, we continue with our intended actions.

In case, the permission is denied,  check with Android if we should show a permission rationale. `shouldShowRequestPermissionRationale` allows us to communicate the purpose of the permission to the user.

`shouldShowRequestPermissionRationale` returns `false` if the user has denied permission with option `Never ask again`. In this case, we can not invoke the Android permission dialog via `requestPermissions` API and user will have to enable permission via app settings. It makes sense to let the user know of the missing permission.

`shouldShowRequestPermissionRationale` Gets whether you should show UI with rationale for requesting permission. You should do this only if you do not have the permission and the context in which the permission is requested does not clearly communicate to the user what would be the benefit of granting this permission.

 

requesting a permission

 

Gotcha: Above implementation only works for Android API level 23 and is not compatible with older versions.

Better Fix: Use `com.android.support:support-v4` as specified in build.gradle line 24. Now your code should use Support library and make the following changes

 

checkSelfPermission(Manifest.permission.WRITE_EXTERNAL_STORAGE)
=> ContextCompat.checkSelfPermission(MainActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE);
requestPermissions(new String[] {Manifest.permission.WRITE_EXTERNAL_STORAGE}, WRITE_EXTERNAL_STORAGE_REQUEST_CODE);
=> ActivityCompat.requestPermissions(MainActivity.this, new String[] {Manifest.permission.WRITE_EXTERNAL_STORAGE}, WRITE_EXTERNAL_STORAGE_REQUEST_CODE);
shouldShowRequestPermissionRationale(Manifest.permission.WRITE_EXTERNAL_STORAGE)
=> ActivityCompat.shouldShowRequestPermissionRationale(MainActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE)

 

And we are done here!!

For targeting Android M or higher, refer to wrapper library EasyPermissions.

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

d3.js Appealing Visualisations

d3.js Appealing Visualisations

d3js appealing visualisations

 

With the ever-increasing amount of data, both in terms of quantity as well as quality, what we need is a precise and accurate way to represent it for better comprehension and facilitate decision making. That’s where d3.js comes to rescue.

d3 stands for Data-Driven Document, i.e. when your web-page is interacting with data. Data can be as simple a simple array of integers or can be as complex as something else.

 

Why choose d3.js?

 

  • it works seamlessly with existing web technologies
  • can manipulate any part of the document object model
  • it is as flexible as the client side web technology stack (HTML, CSS, SVG)
  • takes advantage of built in functionality that the browser has, simplifying the developer’s job, especially for mouse interaction.

 

What d3.js is not?

 

  • it is not a graphics library
  • it is not a data processing library.
  • it doesn’t have pre-built visualizations

 

D3.js is tools that make the connection between data and graphics easy. It sits right between the two, the perfect place for a library meant for data visualization.

 

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation. ~ d3js.org

 

Show me some code…

Simple bar chart

 

<!DOCTYPE html>
<meta charset=utf-8>
<style>
.chart div {
font: 10px sans-serif;
background-color: blue;
text-align: right;
padding: 3px;
margin: 1px;
color: white;
}
</style>
<div class=chart></div>
<script src=http://d3js.org/d3.v3.min.js></script>
<script>
var data_points = [3, 5, 23, 45, 67, 98, 150, 220];
var plot_scale = d3.scale.linear()
.domain(d3.extent(data_points))
.range([5, 420]);
d3.select(.chart)
.selectAll(div)
.data(data_points)
.enter()
.append(div)
.style(width, function(data_point) { return plot_scale(data_point) + px; })
.text(function(data_point) { return data_point; });
</script>
view rawbar_chart.html hosted with ❤ by GitHub
Bar html

 

Well… that’s the only html and does look very nice and professional. We need more power.

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

Temporary Files in Ruby

Temporary Files in Ruby

temporary files in ruby

 

Working with Ruby on Rails applications, many times such as in case of file upload services, generating/processing csv data, uploading data to external services like Amazon there is a need to create temporary files.
A very common solution is to create a usual file object and delete it later. Imagine a scenario where you had created a large data file (say a 2GB) for temporary usage and forgot to delete it.

 

The Solution… Ruby Tempfile Class

 

Tempfile is a ruby utility class for managing temporary files. The class can be used to create temporary files. The file is generated with a unique name each time and is garbage collected when it goes out of scope. This saves you the trouble to have to remove them explicitly.
Since explicitly temporary deleting files is a good idea you can still do it with Tempfile object, Tempfile#unlink .
All actions on a File object are also valid on a Tempfile object, hence no loss of functionality.

Creating a using a temporary file with Tempfile class

 

> tempfile = Tempfile.new([temp, text])
=> #<Tempfile:/tmp/temp20141109-10110-2pjq04text>
> tempfile.write(sample tempfile.)
=> 16
> tempfile.rewind
=> 0
> tempfile.read
=> sample tempfile.
> tempfile.close
=> nil
> tempfile.unlink
=> #<Tempfile:>
view rawtempfile.rb hosted with ❤ by GitHub

 

For complete documentation of class ref Ruby Tempfile

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

gocsv.go Simple CSV parsing with GO

gocsv.go Simple CSV parsing with GO

gocsv go simple csv parsing with go

 

Recently I was working with CSV files in Ruby. Parsing CSV files in Ruby code is easy, thanks to Ruby/csv.

Let’s try it with golang.

Go seems to be the pretty power-packed language for developers. Go, also commonly referred to as golang, is a programming language initially developed at Google in 2007 by Robert Griesemer, Rob Pike, and Ken Thompson. It is a statically-typed language with a syntax loosely derived from that of C, adding garbage collection, type safety, some dynamic-typing capabilities, additional built-in types such as variable-length arrays and key-value maps, and a large standard library. And since it’s from Google, the big giant, Go has built-in support for concurrency with go-routines, channels and select.

Let’s get to work now.

 

//Simple CSV reader
package main
import (
encoding/csv //Package csv reads and writes comma-separated values (CSV) files.
fmt //Package fmt implements formatted I/O with functions analogous to C’s printf and scanf.
io //Package io provides basic interfaces to I/O primitives.
os //Package os provides a platform-independent interface to operating system functionality.
)
// Ref http://golang.org/pkg/ for more on packages and links to documentation
func main() {
//Check for command-line argument filename.
//Ignore additional arguments.
if len(os.Args) < 2 {
fmt.Printf(Error: Source file name is required\n)
fmt.Println(Usage:, os.Args[0], <filename> \n)
return
}
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Println(Error:, err)
return
}
// deferred call to Close() at the end of current method
defer file.Close()
//get a new cvsReader for reading file
reader := csv.NewReader(file)
//Configure reader options Ref http://golang.org/src/pkg/encoding/csv/reader.go?s=#L81
reader.Comma = ; //field delimiter
reader.Comment = # //Comment character
reader.FieldsPerRecord = –1 //Number of records per record. Set to Negative value for variable
reader.TrimLeadingSpace = true
lineCount := 1
for {
// read just one record, but we could ReadAll() as well
record, err := reader.Read()
// end-of-file is fitted into err
if err == io.EOF {
break
} else if err != nil {
fmt.Println(Error:, err)
lineCount += 1
reader.Read()
continue
}
// record is array of strings Ref http://golang.org/src/pkg/encoding/csv/reader.go?s=#L134
fmt.Printf(Record %d: %s\n, lineCount, record)
lineCount += 1
}
}
view raw gocsv.go hosted with ❤ by GitHub

 

Here is a sample CSV file for tests.

Output:

 

 

gocsvgo

 

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.

How to upload a large CSV efficiently using rails!

How to upload a large CSV efficiently using rails!

how to upload a large csv efficiently using rails

 

Active-record is an abstraction layer that facilitates the creation, deletion, and use of ORM objects whose data requires persistent storage to a database. This keeps us away from having to think too much about SQL level queries and makes it very easy for us to work with data. It gives us super easy interface that helps us to do “almost” anything that we can do with bare SQL statements. Apart from the basic CRUD operations, active-record lets us do more complicated database stuff like pick a group of records based on criteria, order them, join tables, perform mathematical operations, etc.

Active-record pattern is liked by most because of the above-mentioned reasons. But using active-record solely may not help when your application scales.

For example, active-record does not have support for bulk insertion. Of course, we can use gems to do that, but I personally do not like the idea of using gem for a specific purpose of the bulk insert. Gems like ActiveRecord-import lets you do that but I wanted to write the code specifically for bulk importing CSV as my requirements did not include using other features of this gem.

Databases like Mysql and Postgres provide native queries to directly import CSV into database tables. Postgres has a “COPY” command for this. But this command requires superuser access to the database, and I did not want to use superuser for CSV import.

We can import CSV records one at a time using active-record but instead, I like to use raw SQL statements in such cases. SQL statements to bulk insert are much faster than active-record way of doing the same task. Let me share an example with you, where using raw SQL statements brought down the CSV import time from several hours to a few seconds.

I am using active-record with rails in my application.

In my application, users are allowed to import large CSV’s from the UI. We require these CSV’s to be imported in the foreground – so background jobs are out of question. This CSV contains only one column – “email”.

Initially, users were uploading CSV’s of not more than 6k rows. But last week, a user tried uploading a file containing 842k records (size: 16MB) and received a timeout. Imagine this situation in a user’s perspective. This will leave the user super confused.

 

The problem – 

 

There were broadly two problems.

I was loading all the data from CSV into the memory and then iterating over it and creating database records one by one using active-record. This made the system to fall apart as the RAM utilisation went up very high.

1
2
3
4
data = CSV.read('emails.csv')
data.each do |e|
  Email.create(email: e)
end

To improve, I rectified one of the biggest mistakes that I was doing i.e. loading all the CSV data into the memory.

 

The Solution – 

 

The solution is to load one record at a time or read in chunks. Obviously, loading single CSV record in the memory and saving it to the database one by one would be very database inefficient, because it would send query to create a record in the database for each CSV row. That means 842k queries to the database. Just for the kicks, I tried it and it was still taking an hour.

1
2
3
CSV.foreach('emails.csv', headers: true) do |row|
  Email.create(email: row[:email])
end

Active-record does not provide support for bulk import. This is one of the reasons why the active-record pattern doesn’t scale very well. So, this time we started writing raw SQL queries for bulk import. We could have done the bulk import for all the 842k records which would have done all the work in not more than 2–3 sec, but raw SQL query would require us to build this query with 842k records in the memory. Hence, for optimal memory consumption, we decided to do it in batches of 5k. So, for every 5k emails, we built the query to bulk insert in the database and as we expected, the time it took to import 842k records in the database was 23sec. (Perhaps 10k batch would bring it down further. This is something I am yet to try). Given the time was acceptable to the user on UI we did not increase.

1
2
3
4
5
6
#For every 5k records in the 'emails' array below
emails = ['a@b.com', 'c@d.com']
email_string = emails.map{|email| "('#{email}')"}.join(',')
#'emails' is table name
query = "insert into emails (email) VALUES #{email_string}"
ActiveRecord::Base.connection.execute(query)

Using this method we could import all the emails in the CSV in 23 seconds but without any validations ( like duplicate emails or blank emails). To avoid duplicate emails I imported these emails into a temporary table and then used database raw queries to copy unique & non-empty records into the actual table.

IMPORTANT: We can use bulk imports, bulk update raw SQL queries carefully wherever we do not require to run the callbacks instantly. Thus, we can move them in the background. Database SQL queries are fast and efficient. In my personal opinion, we should leverage them wherever we can.

I came across this nice blog which has benchmarked timings for various methods to upload CSV using rails. It shows a comparison between 4 different methods to import CSV into the database. The first one being a basic active-record method that takes 210 sec to import 100k records in comparison to importing with SQL validations (This post uses active-record import gem for the same) which brings down this time to 4 sec for the same number of records. Notice that this post used importing with validations. It will take less than 4 sec to import if we import the file without validations.

 

Conclusion

 

Found this blog interesting? Don’t forget to leave your comments and let us know your suggestions.

About RemotePanda

RemotePanda is a personalized platform for companies to hire remote talent and get the quality work delivered from the city Pune. The resources in our talent pool are our close network connections. While connecting them with you, we make sure to manage the quality, growth, legalities, and the delivery of their work. The idea is to make remote work successful for you. Get in touch with us to learn why RemotePanda is the best fit solution for your business requirements.