Recursive Development Teams
True North PHP
Scott Mattocks - November 3, 2012
@scottmattocks - scott@crisscott.com http://joind.in/7425
The whale has nothing to do with this presentation. I just thought it would be more likely to get people out of their lunch coma.
Tweet, blog, email, whatever during my talk. Share ideas. Just loop me in on the conversation.
This conference has a ton of "what I've learned about development" talks.
More than "Use pattern A to solve problem 2."
High level over view of the talk:
Code is written and maintained by teams.
Code can help the team maintain it.
2 independent sites with respectable traffic.
Maybe 100-150K users per day between the two.
9 interconnected sites with over 500K users a day on average
That's just users. Game plays are over 25MM per day.
It isn't facebook level traffic, but they have 1MM developers, we have 6. They have investor money, we have real money.
We can't afford down time.
Talk about me and what we are doing with PHP. A lot of this is drawn from experiences with my team at GSN.
<?php
class Code {
public function maintain() {
$this->fixBugs();
if ($this->bugsExist()) {
$this->maintain();
}
}
}
?>
By the end of the talk, this is what we will end up with. Self maintaining code.
It would be great if this worked. I think there is an RFC in the works for PHP 6.
What we are going after, isn't too far from this.
We want code that helps take care of itself.
Writing code is easy
<?php
$tweet = 'Hello Twitter';
$result = post_tweet($tweet);
function post_tweet($tweet) {
$conn = new OAuth(array('consumer_key' => '******',
'consumer_secret' => '******', 'user_token' => '******',
'user_secret' => '******'));
$conn->request('POST', $conn->url('1/statuses/update'),
array('status' => $tweet));
return $conn->response['code'];
}
?>
This posts stuff to Twitter
Thanks, 140dev.com
Writing code that does stuff is not that hard. Put the right words in the right order.
As Reg pointed out yesterday, PHP is pretty frictionless. Taking things from an idea to an implementation is pretty quick.
One person can do some pretty neat things with code.
The slow and painful part comes when you have to keep your code running and keep your customers happy.
Talk about the process of finding bugs and fixing stuff.
First you have to know there is a bug.
How do you know something went wrong? Customer complaints? A drop is sales?
How do you find the bug, or the piece of the system that failed? A debugger?
How do you make sure the bug is fixed? Hopefully, unit tests.
There is more to this than just fixing bugs. The internet is unreliable.
If your software relies on other software, you will have a problem eventually.
Life is better when your application is more stable and easier to fix.
Thanks, Dunechaser
That's the Confoo organizers beating up on Chris after he tweeted about rejection emails. I can say that because they already booked my flight.
Working as part of a team makes things easier.
PHP has a huge team. The PHP community.
Think back to some of the characteristics of the better software teams you have worked with
Working with teams is great for getting a lot of stuff done quickly but teams mean you are working with people of different skill levels.
I was speaking to Larry Ullman on Thursday and he said roughly "Bugs are learning opportunities. Bad developers aren't really bad developers. They are really just better at providing learning opportunities."
That may be great for helping people learn. It isn't so great for making money.
Teams make it easy
Good teams...
write things down
know what to expect from each other
adapt to changes
don't let bad things bring them down
communicate clearly
They send emails. They update tickets. They make meaningful commit messages.
They make commitments and live up to them. If they say that they are going to do something they do it.
They deal with changing priorities somewhat professionally. No body likes it but it happens (even in scrum teams).
There is always someone who is a dick. Good teams work well inspite of him/her.
Clear communication isn't just putting observations into words. It is also about explaining intent and purpose.
Software is meant to do what people do (only faster or more concurrently than a person can)
It isn't just that good developers do these things. They do them wihtout thinking and without being asked.
You don't have to stay on top of these people and force them to do things this way. They know that these things are important and they just do them.
The feedback loop needs to be small and quick. When you ask them what went wrong, they can get the information back to you quickly.
Who's on your team?
Hint: if you didn't answer "my code" something is wrong
Is that all it does?
The more code you write to support the number crunchers, the less time you spend trying to figure out what the crunchers are doing
I am not advocating that you violate the single responsibility principle.
Your user class should only be publicly responsible for dealing with users. However, it also needs to let the team know what it is doing.
Good team members write things down
Good code logs things
Developers update tickets in the bug tracker.
They add commit messages.
They email each other when they are done thing or when things are going well or when things are going really well.
fragment
A ⇒ B ⇒ C
Code is just a rough guideline for data to follow.
Data doesn't always listen. Like a shady salesman, it slips through cracks and changes its appearance on the fly.
Data moves through the system like a game of telephone.
Logging is not just keeping track of which requests were made
You need to track how data moves through the system
How does it change as it goes from method to method?
You have a plan for your data, but a small bug can send it in a radically different direction
This isn't logging
<?php
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
die('Connection failed: ' . $e->getMessage());
}
?>
I'm talking to you PDO::__construct() docs!
Did something make it to a log file somewhere? Probably.
Is it all that helpful? Probably not. If you see this message in a log file at 3 AM are you going to be able to fix this problem quickly?
Is someone else going to be able to fix it quickly?
This is better
<?php
try {
log('Connecting to the database ' . $dsn);
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
log('Connection failed to ' . $dsn . ': ' . $e->getMessage());
exit;
}
?>
Assuming the log function tracks some context for us.
This is pretty good
public function register($usernmae, $password) {
$this->log->trace('Entering ' . __CLASS__ . '::' . __FUNCTION__ ' with username = ' . $username . ' and password = ' . str_repeat('*', strlen($password)));
if (!$this->_validateUsername($username)) {
$this->log->warn('Invalid username when trying to register a new user: ' . $username);
$this->log->trace('Exiting ' . __CLASS__ . '::' . __FUNCTION__ . ' on error.');
return;
}
/* Validate the password and do the registration */
$this->log->info('New user registered with username ' . $username . ' and user ID ' . $this->_id);
$this->log->trace('Exiting ' . __CLASS__ . '::' . __FUNCTION__ . ' on success.');
return;
}
I am constrained to one slide. I might have added more logging. I certainly would have added some comments.
There is a line that says we entered the method. It doesn't just say "we got here" it says we got here with specific data.
Every time we leave the method, there is a log entry. It says why we left, not just that we left.
When something went wrong, we threw a warning. It says, "something unexpected happened, but we can continue."
We have an info line that lets us keep track of interesting events. Good for analytics.
Remember to keep user information confidential. Don't compromise security for data clarity.
If all your logs do is tell you that you have a bug and help you fix that bug, you are missing a very important piece.
You need to know which user had a problem, and how you can correct their data.
"Failed changing username from 'scott' to 'bieberfan4ever' for user ID 8675309"
Sometimes you need a firehose
Trace - method by method, condition by condition.
Debug - If you put something in while writing the code, leave it there.
Info - Used to inform someone or something about a notable event.
Warn - Something unexpected happened, but we can recover.
Error - Something unexpected happened and we can't recover.
Get the right amount of information at just the right time.
When logging is set to trace, you should be able to rebuild your code just from the logs.
You can't have trace on all the time. It will fill up your disk quickly.
"We can recover" means we can get information back to the user, not we can do what the user asked.
Failure to connect to _a_ database should be a warning.
Failure to connect to _all_ databases is an error.
A user entering the wrong password does not need to be logged.
Track down Hugo Hammon for info about MonoLog
Good developers don't just tell you what they are doing now; they tell you what they are going to do.
Good code predicts the future using unit tests.
Test Driven Development is great. Unit tests are great. There are lots of reasons to write unit tests.
I'm not here to talk about how to write unit tests. Go see Chris for that.
Good unit tests are crucial to having a well functioning system.
Unit tests answer questions like "What happens if I try to register a user with the name 'bieberrulz'?"
The more questions you can answer, the better you are at predicting the future.
300 unit tests !== 1 application
What is the purpose of unit tests? The goal is to make sure that all of the individual pieces are working so that you have greater confidence that they will work correctly when strung togethr.
Data goes into a method in one form, and comes out in another. The data is then fed into another method.
Each link in a chain is useful by itself, but they are much more useful when they are all hooked together.
Unit tests should be less like a pile of unlinked chain pieces and more like a set of directions.
Each test is like a map tile.
Zoom out and you see how to get from A to B.
String the tests together using real world data.
Meh.
<?php
// Test set up stuff and mocking goes here.
$this->assertEquals($validator->validateEmail('bob@example.com'));
?>
Less Meh.
<?php
// Test set up stuff and mocking goes here.
$this->assertTrue($validator->validateEmail('scott.mattocks@gmail.com'));
?>
The difference is small, but significant.
This caught us at GSN. We had validation issues with . in Gmail addresses. Talk about why and how we failed to catch this.
Using real world data, pulled from logs, makes your tests more reliable in a real world scenario.
Don't use Faker
LB Denker talked about the Four Tenets of Testing:
1. Know how much is being covered - you won't have 100% coverage. Use your logs to figure out where you really need coverage.
2. Know what is being tested - Using real world data means you are testing the real world
Good teams...
write things down (Logging)
know what to expect from each other (Unit Tests)
Good developers adapt quickly to changes.
Good code changes its behavior without changing the code.
Being part of a team means listening to what others have to say and chaning how you do things.
LB Denker says, "You can't make a developer do anything." I would modify that to say, "You can't make a bad developer do anything."
With develoeprs we expect them to take direction - "Do things this way" or "You might want to change how you are looping through that array"
Code shouldn't change behavior abitrarily, it listens to changes in configurations.
Code needs to change, because the real world isn't static.
Dev
⇒
QA
⇒
Staging
⇒
Produciton
$> svn ls $SVNROOT/awesomeproject/trunk
application/
bin/
conf/
doc/
lib/
tests/
Even if you don't change server infrastructre, you are probably working with different infrastructures every day.
You expect your development teams to behave differently in the office, than you do when they work from home or when they are getting some beers after work.
Code should function differently depending on where it is doing its work.
Adam Goucher - Turn off third party integrations like facebook and twitter in your automation environment.
If we launch with a cloud based solution, we can change the configs to be mroe fault tolerant.
If we move to a more reliable set up, we don't have to make code changes. We just change the configuraitons to fail more quickly.
Realistically, it is more likely to move the other way. From a stable environment to a less stable environment.
In dev and QA, you want it to fail quickly and display errors differently.
You should have different configuraitons for different environments.
Configuraitons should not be checked into the source code repo.
Maybe you can have defaults.
If you aren't the person responsible for the produciton database, you shouldn't now the password.
Knowing the password means you are a suspect when something bad happens.
My application is database driven. If we can't connect to the database, we have bigger problems.
— Me (1978 - 2011)
Configuration options aren't just for different environments. As with the cloud vs. hosted example, expectations change.
Config flags and values allow you to alter the flow of code without altering the code directly.
That was me most of my career.
You may have bigger problems, but does the user really care? They just want to give you money and play games.
If a developer came up to you and said, "I tried writing a unit test but I couldn't figure it out so I gave up," what would you tell them?
Just because you can't connect to a database doesn't mean you can't connect to a database.
The number of times you try to connect to a database needs to be controlled by conrigurations.
The number of databases you try to connect to needs to be controlled by configuration.
<?php
for (i = 0; $i < $config['db_primary_tries'], ++$i) {
try {
$db = new Database($config['db_host'], $config['db_user'], $config['db_password'])
break;
} catch (DBException $dbe) {
$log->warn('Failed connecting to primary database ' . $config['db_host'] . ' ' . ($i + 1) . ' times. ' . $dbe->getMessage());
usleep($config['db_retry_sleep']);
}
}
if (empty($db)) {
$log->error('Could not connect to primary databse ' . $config['db_host'] . ' after ' . $config['db_primary_tries'] . ' attempts. Trying secondary.');
}
// Rinse and repeat for the secondary.
?>
The amount of effort we put into trying to connect to the database doesn't have to be hard coded.
Talk about GSN nodes. How we moved stuff to a cloud of sorts and how we needed to change our expectations.
Good teams...
write things down (Logging)
know what to expect from each other (Unit Tests)
adapt quickly to changes (Configurations)
Good developers don't let bad developers bring them down.
Good code isolates features from each other.
Jeff Carter was on the Flyers. He was a great player. He was a jerk. They got rid of him. They were better for it.
He won the stanley cup with LA last year.
Good players can be bad for your team.
Don't let failures in one segment of the architecture break your entire application.
Cache
⇔
Secondary DB
⇔
Primary DB
Expect failure, but don't accept it.
When one developer is out sick, on vacation, or at a conference outside Toronto, does all work stop?
Good teams need to be able to work around missing pieces.
An application that is built for performance, probably has a caching layer in front of multiple databases.
If not, go track down Ilia and ask him about caching.
The cache is very fast. All it does is look up values based on keys.
The secondary (aka read only) database should be optimized for reading. You shouldn't write to it.
The primary database should handle all writes.
If you are using a write through cache, your application only talks to the caching layer. - Talk to Rich from WattPad
The caching layer probably isn't a hard and fast requirement for your system. It is designed as a performance booster. If it goes down it probably isn't a big deal right?
What will your application do if the cache fails?
We know it will log some very useful and helpful messages that will trigger monitoring software and whatnot. But what about the customer?
Hopefully, it is more than just get sad and present an error message.
If it isn't there just go around it.
The secondary database is probably along the same lines as the cache.
What about the primary db? Can your application withstand a failure there?
If you have good logging, you can probably get away with much more than you think.
If a connection fails, automatically change the logging level to start capturing enough information to be able to write to the DB later.
Continue reading from the cache and secondary DB and provide the user with the best experience you can.
How to connect to a database or any other system
Try to connect using the configured timeout settings
Log failure as a warning
Retry the configured number of times, waiting the configurable amount of times between each
Try the configured back up
Repeat 2 & 3 if unsuccessful, log success as a warning
If no connections can be made successfully, log an error, but don't halt execution (it is up to the application to figure out if execution can continue)
Don't give up.
Try other sources.
Log the crap out of stuff.
That last one there is important.
Your DB abstraction layer is not the authority on what your application can and cannot do without a database conneciton.
This means that you have to have a bit more logic (or gasp! repeated code) in order to have an application that keeps running.
You don't give someone production access the day you hire them.
Don't release features the instant you deploy the code.
If you take out the bad code, feature, thrid party, etc., you can relax a bit while fixing the problem.
Talk about the old GSN release process. Why it was no good and how much pain it caused.
Then talk about the new process.
If you can turn a feature on after the code is deployed, you can probably turn it off too.
If your system is reasonably complex, it probably has several features.
Does a bug in your "tag a photo" feature cause trouble for your post a photo feature?
When a kid in class is causing trouble, you make him sit in the corner.
Being able to remove a feature through configurations means you can immediately quarantine misbehaving code.
When you can isolate features and quickly remove them from the user experience, you can deploy code with significantly less risk.
A good A/B testing framework takes care of some of this for you. If A is sufficiently isolated from B, you can just make the A group be 100% of traffic.
Gigya sucks. When it goes down, we can turn it off. It would be great if we did this automatically based on log data.
Good teams...
write things down (Logging)
know what to expect from each other (Unit Tests)
adapt quickly to changes (Configurations)
don't let bad things bring them down (Isolation)
Good developers communicate well with non-technical folks.
Good code doesn't require you to read the code in order to know what the code is doing.
Explaining how the system is intended to work is a part of every developer's job.
There are always limitations to the software.
"If we let you schedule stuff for arbitrary times, the admin interface for scheduling promos would have been a mess and you likely would have messed things up multiple times a day which means we would have to manually update the database putting the entire system at risk."
"Because of the concessions we had to make for usability, all of your promotions need to start at the top of the hour."
Let that sink in for a minute. Good code makes good code unnecessary.
<?php
public function testWebDriver() {
$home_page = new Home(self::$session);
$home_page->open();
$schedule_page = $home_page->navigate_to('Schedule');
$schedule_page->open_session('WebDriver!');
$this->assertEquals($schedule_page->title, "WebDriver!");
}
?>
Adam Goucher - WebDriver! - "We create a page and then we open a page because creating it doesn't open it."
Matt Frost posted recently about the necessity of documentation. - http://shortwhitebaldguy.com/blog/2012/10/using-comments
Antonin Januska - http://antjanus.com/blog/web-design-tips/best-comment-separator/
Even simple code needs comments.
Every speaker that showed code (including myself) doesn't just show the code. They spend time explaining what it does.
If code as easy to read and understand, comments wouldn't exist and speakers would just be able to show the slides without talking.
<?php
/**
* Returns the array of AMQP arguments for the given queue.
*
* Depending on the configuration available, we may have one or more arguments which
* need to be sent to RabbitMQ when the queue is declared. These arguments could be
* things like high availability configurations.
*
* If something in getInstance() is failing, check here first. Trying to declare a
* queue with a set of arguments that does not match the arguments which were used
* the first time the queue was declared most likely will not work. Check the config
* for AMQP and make sure that the arguments have not been changed since the queue
* was originally created. The easiest way to reset them is to kill off the queue
* and try to recreate it based on the new config.
*
* @param string $name The name of the queue.
*
* @return array The array of arguments from the config.
*/
?>
Start with the documentation. Write out what you are doing as a stroy.
Write in line comments that explain exactly what each block is doing. Even if they are redundant, you have something that is easier to read and understand.
There are other people that are responsible for maintaining your application.
If the Ops guy can read the code and see what you wanted it to do without having to tax his/her brain at 3 AM, you are much more likely to get a good night's sleep.
Not everyone knows what array_walk does. Not everyone can tell if you are using it correctly.
The keys is to go from issue to resolution as quickly as possible.
If you weren't in Rafael's presentation on readability, go review his slides. Just ignore some of the last rule.
<?php
private static function _getQueueArgs($name)
{
// Start with nothing.
$args = array();
// We may need to set some configuration arguments.
$cfg = Settings\AMQP::getInstance();
// Check for queue specific args first and then try defaults.
if (array_key_exists($name, $cfg['queue_arguments'])) {
$args = $cfg['queue_arguments'][$name];
} elseif (array_key_exists('default', $cfg['queue_arguments'])) {
$args = $cfg['queue_arguments']['default'];
}
return $args;
}
?>
This is not a talk on dependency injection.
Here's the code for it.
Good teams...
write things down (Logging)
know what to expect from each other (Unit tests)
adapt quickly to changes (Configurations)
don't let bad things bring them down (Isolation)
communicate clearly (Documentation)
Every team has a jerk. Every team has someone that thinks they can do better and wants to do things their own way. If you don't understand their stuff, you are an idiot.
Don't let your code be the jerk. You code is the one team member you can change.
We're Hiring!
We're Everywhere
Waltham, MA
Washington, D.C.
San Francisco, CA
We sponsor visas
Great Benefits
Unlimited vacation and sick time
401K matching
Hack day every 5 weeks
No Such Thing as Routine
DAUs measures in millions
Web, Mobile, and Social
PHP, Perl, Unity, Objective-C, Java, MongoDB…