There is nothing bad about living in the province instead of the big city (though a lot of people might disagree), but there certainly are a couple of challenges you'll have to deal with. Whether you're a rugged outdoorsperson, a cozy homebody, or an urban explorer, living in the inaka involves having to deal with being far from the city's conveniences, a slightly tougher language and cultural barrier, and over-all isolation, among other things. However, don't let that get your hopes down. There certainly are ways to conquer those challenges.
First of all, isolation is probably going to be what you'd like to tackle immediately. No matter how much of an introvert someone will proclaim to be, the truth is that no person is an island. You'll probably want to at least befriend someone so you can feel comfortable around them; make a safe place, so to speak.
If your city or town has an Irish pub (or any popular hub for foreigners), it's definitely a good idea to hit it up and try to mingle with the locals. Sometimes, there will be Japanese people interested in chatting in English. You can also ask people you work with about other popular places in the area. If you feel pretty confident in your conversational Japanese, becoming a regular at small shops is an offbeat option.
Aside from going to shops and restaurants to connect with other people, there's also the option of finding associations and other similar groups. Belonging to a religious group might sometimes work to your favor. You might be surprised to find your denomination's place of worship within the smallest of provinces. If you aren't religious, though, there are also other options. Civic Centers with a department for International Relations (often also providing free Japanese lessons for beginners) may be the best place to start. Also, if you're lucky, try checking out Meetup (http://www.meetup.com). You might be surprised to find an interesting group open to foreigners in your neck of the woods.
Once you've found people to connect with, you'll probably want to find ways to get the things you need easily. Often, the countryside makes it pretty difficult to get to shopping malls and other places to get things. Also, shops in the countryside also have a more limited selection. Thankfully, internet shopping is a big thing in Japan! COD and convenience store payments also make it really easy to buy things without having to use credit. In case you need a credit card but don't want to use the one from your home country or don't have one in the first place, there are prepaid options like the V-Preca from Lifecard (http://vpc.lifecard.co.jp/en/). Paired with Japan's very reliable courier service, you won't have to worry much about needing to go the distance for the things you want or need.
Finally, once you're all settled, you might have to deal with the problem of boredom. Living in the countryside always means having less happening. It's not as exciting as being in a city. With that said, it's very important to give yourself things to do. As a homebody, I'm a big fan of picking up hobbies. And Japan is a really amazing place for hobbyists. Supplies for a wide variety of hobbies can be bought at 100 yen shops. For more specialized hobbies such as model building, you can get all kinds of tools and kits at toy stores and sometimes even appliance shops.
Not a fan of staying indoors? Pick up a sport or look into interesting new things happening in your prefecture. As Japan is constantly trying to pick up its tourism, there's bound to be interesting gimmicks happening on your local train line or in nearby cities. Just keep an eye out for announcements around your home station. Who knows, you might discover a whole new side of Japan not many foreigners get to see!